2024-04-26T21:17:04.512 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-26T21:17:04.512 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} --name rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi --owner scheduled_rishabh@teuthology --archive /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675215 --verbose -- /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675215/orig.config.yaml 2024-04-26T21:17:04.564 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T21:17:04.657 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675215 branch: wip-rishabh-testing-20240426.111959 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} email: null first_in_suite: false job_id: '7675215' kernel: client: sha1: distro flavor: default kdb: true sha1: 806ec36b21129f113ce356c007dd59694d340b37 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-rishabh-testing-20240426.111959 ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-testing-20240426.111959 sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 owner: scheduled_rishabh@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 816 sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 sleep_before_teardown: 0 subset: 166/512 suite: fs suite_branch: wip-rishabh-testing-20240426.111959 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi046.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCPitY8pbKyqnD3letyL81Rd7976Rf9F5cF0n8DCQvpjF5yIjE8lfMg/4hlR0BEwRKWZhD8Q3ZPYahzPkbuVOT8= smithi050.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDKRkScLIIgerq406FE4WuJpgbkzbP5Zy3nIcvvak+ZGp4USLJ0Z62WLUR0fFPmieX3Tr0lxlVADn+tqGdyh8FY= smithi142.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDCpk9CFgwBtbGIYebYdy+7xw10lkXXTiTBbhGZb8esOlDP4FDWEYDXifKfcaGi7jsJKmgJOrGfCuaF75MGtUvQ= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - workunit: clients: all: - suites/fsstress.sh timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-26_19:30:57 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-26T21:17:04.658 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa; will attempt to use it 2024-04-26T21:17:04.658 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa/tasks 2024-04-26T21:17:04.658 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-26T21:17:04.660 INFO:teuthology.task.internal:Checking packages... 2024-04-26T21:17:04.684 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bc612ffb9c1d5bc27536bc917a705262a36d4387' 2024-04-26T21:17:04.684 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-26T21:17:04.684 INFO:teuthology.packaging:ref: None 2024-04-26T21:17:04.684 INFO:teuthology.packaging:tag: None 2024-04-26T21:17:04.684 INFO:teuthology.packaging:branch: wip-rishabh-testing-20240426.111959 2024-04-26T21:17:04.684 INFO:teuthology.packaging:sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T21:17:04.684 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-rishabh-testing-20240426.111959 2024-04-26T21:17:04.824 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3306.gbc612ffb 2024-04-26T21:17:04.825 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-26T21:17:04.836 INFO:teuthology.task.internal:no buildpackages task found 2024-04-26T21:17:04.836 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-26T21:17:04.854 INFO:teuthology.task.internal:Saving configuration 2024-04-26T21:17:04.871 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-26T21:17:04.882 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T21:17:04.907 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675215', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 21:11:56.421096', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCPitY8pbKyqnD3letyL81Rd7976Rf9F5cF0n8DCQvpjF5yIjE8lfMg/4hlR0BEwRKWZhD8Q3ZPYahzPkbuVOT8='} 2024-04-26T21:17:04.926 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675215', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 21:11:56.418044', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDKRkScLIIgerq406FE4WuJpgbkzbP5Zy3nIcvvak+ZGp4USLJ0Z62WLUR0fFPmieX3Tr0lxlVADn+tqGdyh8FY='} 2024-04-26T21:17:04.946 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi142.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675215', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 21:11:56.419723', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDCpk9CFgwBtbGIYebYdy+7xw10lkXXTiTBbhGZb8esOlDP4FDWEYDXifKfcaGi7jsJKmgJOrGfCuaF75MGtUvQ='} 2024-04-26T21:17:04.946 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-26T21:17:04.958 INFO:teuthology.task.internal:roles: ubuntu@smithi046.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-26T21:17:04.958 INFO:teuthology.task.internal:roles: ubuntu@smithi050.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-26T21:17:04.958 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-26T21:17:04.958 INFO:teuthology.run_tasks:Running task console_log... 2024-04-26T21:17:05.107 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2f2c6da160>, signals=[15]) 2024-04-26T21:17:05.108 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-26T21:17:05.118 INFO:teuthology.task.internal:Opening connections... 2024-04-26T21:17:05.118 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi046.front.sepia.ceph.com 2024-04-26T21:17:05.120 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T21:17:05.204 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi050.front.sepia.ceph.com 2024-04-26T21:17:05.205 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T21:17:05.279 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi142.front.sepia.ceph.com 2024-04-26T21:17:05.280 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T21:17:05.352 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-26T21:17:05.361 DEBUG:teuthology.orchestra.run.smithi046:> uname -m 2024-04-26T21:17:05.379 INFO:teuthology.orchestra.run.smithi046.stdout:x86_64 2024-04-26T21:17:05.379 DEBUG:teuthology.orchestra.run.smithi046:> cat /etc/os-release 2024-04-26T21:17:05.438 INFO:teuthology.orchestra.run.smithi046.stdout:NAME="CentOS Stream" 2024-04-26T21:17:05.438 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION="9" 2024-04-26T21:17:05.438 INFO:teuthology.orchestra.run.smithi046.stdout:ID="centos" 2024-04-26T21:17:05.438 INFO:teuthology.orchestra.run.smithi046.stdout:ID_LIKE="rhel fedora" 2024-04-26T21:17:05.439 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_ID="9" 2024-04-26T21:17:05.439 INFO:teuthology.orchestra.run.smithi046.stdout:PLATFORM_ID="platform:el9" 2024-04-26T21:17:05.439 INFO:teuthology.orchestra.run.smithi046.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T21:17:05.439 INFO:teuthology.orchestra.run.smithi046.stdout:ANSI_COLOR="0;31" 2024-04-26T21:17:05.439 INFO:teuthology.orchestra.run.smithi046.stdout:LOGO="fedora-logo-icon" 2024-04-26T21:17:05.439 INFO:teuthology.orchestra.run.smithi046.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T21:17:05.439 INFO:teuthology.orchestra.run.smithi046.stdout:HOME_URL="https://centos.org/" 2024-04-26T21:17:05.439 INFO:teuthology.orchestra.run.smithi046.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T21:17:05.439 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T21:17:05.440 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T21:17:05.440 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2024-04-26T21:17:05.518 DEBUG:teuthology.orchestra.run.smithi050:> uname -m 2024-04-26T21:17:05.537 INFO:teuthology.orchestra.run.smithi050.stdout:x86_64 2024-04-26T21:17:05.537 DEBUG:teuthology.orchestra.run.smithi050:> cat /etc/os-release 2024-04-26T21:17:05.596 INFO:teuthology.orchestra.run.smithi050.stdout:NAME="CentOS Stream" 2024-04-26T21:17:05.596 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION="9" 2024-04-26T21:17:05.596 INFO:teuthology.orchestra.run.smithi050.stdout:ID="centos" 2024-04-26T21:17:05.596 INFO:teuthology.orchestra.run.smithi050.stdout:ID_LIKE="rhel fedora" 2024-04-26T21:17:05.597 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_ID="9" 2024-04-26T21:17:05.597 INFO:teuthology.orchestra.run.smithi050.stdout:PLATFORM_ID="platform:el9" 2024-04-26T21:17:05.597 INFO:teuthology.orchestra.run.smithi050.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T21:17:05.597 INFO:teuthology.orchestra.run.smithi050.stdout:ANSI_COLOR="0;31" 2024-04-26T21:17:05.597 INFO:teuthology.orchestra.run.smithi050.stdout:LOGO="fedora-logo-icon" 2024-04-26T21:17:05.597 INFO:teuthology.orchestra.run.smithi050.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T21:17:05.597 INFO:teuthology.orchestra.run.smithi050.stdout:HOME_URL="https://centos.org/" 2024-04-26T21:17:05.597 INFO:teuthology.orchestra.run.smithi050.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T21:17:05.597 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T21:17:05.597 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T21:17:05.597 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server 2024-04-26T21:17:05.617 DEBUG:teuthology.orchestra.run.smithi142:> uname -m 2024-04-26T21:17:05.635 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64 2024-04-26T21:17:05.635 DEBUG:teuthology.orchestra.run.smithi142:> cat /etc/os-release 2024-04-26T21:17:05.692 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="CentOS Stream" 2024-04-26T21:17:05.692 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="9" 2024-04-26T21:17:05.692 INFO:teuthology.orchestra.run.smithi142.stdout:ID="centos" 2024-04-26T21:17:05.692 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE="rhel fedora" 2024-04-26T21:17:05.692 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="9" 2024-04-26T21:17:05.692 INFO:teuthology.orchestra.run.smithi142.stdout:PLATFORM_ID="platform:el9" 2024-04-26T21:17:05.692 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T21:17:05.692 INFO:teuthology.orchestra.run.smithi142.stdout:ANSI_COLOR="0;31" 2024-04-26T21:17:05.692 INFO:teuthology.orchestra.run.smithi142.stdout:LOGO="fedora-logo-icon" 2024-04-26T21:17:05.693 INFO:teuthology.orchestra.run.smithi142.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T21:17:05.693 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://centos.org/" 2024-04-26T21:17:05.693 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T21:17:05.693 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T21:17:05.693 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T21:17:05.693 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server 2024-04-26T21:17:05.709 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-26T21:17:05.782 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-26T21:17:05.844 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-26T21:17:05.844 DEBUG:teuthology.orchestra.run.smithi046:> test '!' -e /home/ubuntu/cephtest 2024-04-26T21:17:05.846 DEBUG:teuthology.orchestra.run.smithi050:> test '!' -e /home/ubuntu/cephtest 2024-04-26T21:17:05.848 DEBUG:teuthology.orchestra.run.smithi142:> test '!' -e /home/ubuntu/cephtest 2024-04-26T21:17:05.865 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-26T21:17:05.872 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-26T21:17:05.872 DEBUG:teuthology.orchestra.run.smithi046:> test -z $(ls -A /var/lib/ceph) 2024-04-26T21:17:05.904 DEBUG:teuthology.orchestra.run.smithi050:> test -z $(ls -A /var/lib/ceph) 2024-04-26T21:17:05.906 DEBUG:teuthology.orchestra.run.smithi142:> test -z $(ls -A /var/lib/ceph) 2024-04-26T21:17:06.019 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-26T21:17:06.092 INFO:teuthology.run_tasks:Running task kernel... 2024-04-26T21:17:06.109 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'} 2024-04-26T21:17:06.109 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-26T21:17:06.109 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-26T21:17:06.109 DEBUG:teuthology.orchestra.run.smithi046:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T21:17:06.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T21:17:06.127 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2024-04-26T21:17:06.184 INFO:teuthology.orchestra.run.smithi046.stdout:5.14.0-437.el9.x86_64 2024-04-26T21:17:06.185 INFO:teuthology.task.kernel:Running kernel on smithi046: 5.14.0-437.el9.x86_64 2024-04-26T21:17:06.185 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum install -y kernel 2024-04-26T21:17:08.991 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - BaseOS 7.0 MB/s | 8.0 MB 00:01 2024-04-26T21:17:11.964 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - AppStream 10 MB/s | 19 MB 00:01 2024-04-26T21:17:17.423 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.2 MB 00:01 2024-04-26T21:17:18.991 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 16 kB 00:00 2024-04-26T21:17:21.628 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 8.3 MB/s | 21 MB 00:02 2024-04-26T21:17:26.900 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-26T21:17:27.087 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:01 ago on Fri 26 Apr 2024 09:17:26 PM UTC. 2024-04-26T21:17:28.840 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-26T21:17:28.840 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-26T21:17:28.928 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-26T21:17:28.929 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-04-26T21:17:28.929 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-26T21:17:29.034 DEBUG:teuthology.orchestra.run.smithi046:> echo no | sudo yum reinstall kernel || true 2024-04-26T21:17:29.554 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:03 ago on Fri 26 Apr 2024 09:17:26 PM UTC. 2024-04-26T21:17:29.682 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-26T21:17:29.698 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-26T21:17:29.698 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2024-04-26T21:17:29.698 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-26T21:17:29.698 INFO:teuthology.orchestra.run.smithi046.stdout:Reinstalling: 2024-04-26T21:17:29.698 INFO:teuthology.orchestra.run.smithi046.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-26T21:17:29.699 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:17:29.699 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-04-26T21:17:29.699 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-26T21:17:29.699 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:17:29.699 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 255 k 2024-04-26T21:17:29.699 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 0 2024-04-26T21:17:29.699 INFO:teuthology.orchestra.run.smithi046.stderr:Operation aborted. 2024-04-26T21:17:29.760 INFO:teuthology.orchestra.run.smithi046.stdout:Is this ok [y/N]: 2024-04-26T21:17:29.760 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum reinstall -y kernel || true 2024-04-26T21:17:30.276 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:04 ago on Fri 26 Apr 2024 09:17:26 PM UTC. 2024-04-26T21:17:30.396 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-26T21:17:30.397 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-26T21:17:30.397 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2024-04-26T21:17:30.397 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-26T21:17:30.397 INFO:teuthology.orchestra.run.smithi046.stdout:Reinstalling: 2024-04-26T21:17:30.397 INFO:teuthology.orchestra.run.smithi046.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-26T21:17:30.397 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:17:30.397 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-04-26T21:17:30.398 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-04-26T21:17:30.398 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:17:30.398 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 255 k 2024-04-26T21:17:30.398 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 0 2024-04-26T21:17:30.398 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-04-26T21:17:30.635 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.5 MB/s | 255 kB 00:00 2024-04-26T21:17:30.636 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-04-26T21:17:30.636 INFO:teuthology.orchestra.run.smithi046.stdout:Total 1.0 MB/s | 255 kB 00:00 2024-04-26T21:17:30.656 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-04-26T21:17:30.840 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-04-26T21:17:30.840 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-04-26T21:17:30.845 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-04-26T21:17:30.845 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-04-26T21:17:30.975 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-04-26T21:17:31.053 INFO:teuthology.orchestra.run.smithi046.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-26T21:17:31.053 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-26T21:17:31.763 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-26T21:17:31.763 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-26T21:17:32.485 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-26T21:17:32.485 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:17:32.485 INFO:teuthology.orchestra.run.smithi046.stdout:Reinstalled: 2024-04-26T21:17:32.486 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-26T21:17:32.486 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:17:32.486 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-26T21:17:32.658 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q kernel | sort -rV | head -n 1 2024-04-26T21:17:32.706 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-26T21:17:32.706 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-26T21:17:32.706 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-26T21:17:32.706 DEBUG:teuthology.parallel:result is None 2024-04-26T21:17:32.707 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-26T21:17:32.713 INFO:teuthology.task.internal:Creating test directory... 2024-04-26T21:17:32.713 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T21:17:32.716 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T21:17:32.718 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T21:17:32.736 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-26T21:17:32.744 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-26T21:17:32.791 INFO:teuthology.task.internal:Creating archive directory... 2024-04-26T21:17:32.791 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T21:17:32.794 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T21:17:32.796 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T21:17:32.828 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-26T21:17:32.836 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-26T21:17:32.836 DEBUG:teuthology.orchestra.run.smithi050:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T21:17:32.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T21:17:32.879 DEBUG:teuthology.orchestra.run.smithi142:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T21:17:32.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T21:17:32.895 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T21:17:32.897 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T21:17:32.922 DEBUG:teuthology.orchestra.run.smithi142:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T21:17:32.928 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T21:17:32.947 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T21:17:32.958 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T21:17:32.968 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T21:17:32.978 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T21:17:32.990 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T21:17:32.991 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-26T21:17:32.998 INFO:teuthology.task.internal:Configuring sudo... 2024-04-26T21:17:32.998 DEBUG:teuthology.orchestra.run.smithi046:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T21:17:33.001 DEBUG:teuthology.orchestra.run.smithi050:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T21:17:33.023 DEBUG:teuthology.orchestra.run.smithi142:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T21:17:33.057 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-26T21:17:33.066 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-26T21:17:33.066 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T21:17:33.069 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T21:17:33.090 DEBUG:teuthology.orchestra.run.smithi142:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T21:17:33.113 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T21:17:33.158 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T21:17:33.216 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T21:17:33.297 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T21:17:33.336 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-26T21:17:33.337 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T21:17:33.400 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T21:17:33.443 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T21:17:33.488 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T21:17:33.569 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T21:17:33.611 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T21:17:33.611 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T21:17:33.677 DEBUG:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T21:17:33.722 DEBUG:teuthology.orchestra.run.smithi142:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T21:17:33.777 DEBUG:teuthology.orchestra.run.smithi142:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T21:17:33.861 DEBUG:teuthology.orchestra.run.smithi142:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T21:17:33.902 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-04-26T21:17:33.902 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T21:17:33.968 DEBUG:teuthology.orchestra.run.smithi046:> sudo service rsyslog restart 2024-04-26T21:17:33.970 DEBUG:teuthology.orchestra.run.smithi050:> sudo service rsyslog restart 2024-04-26T21:17:33.972 DEBUG:teuthology.orchestra.run.smithi142:> sudo service rsyslog restart 2024-04-26T21:17:34.004 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T21:17:34.013 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T21:17:34.044 INFO:teuthology.orchestra.run.smithi142.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T21:17:34.407 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-26T21:17:34.414 INFO:teuthology.task.internal:Starting timer... 2024-04-26T21:17:34.414 INFO:teuthology.run_tasks:Running task pcp... 2024-04-26T21:17:34.459 INFO:teuthology.run_tasks:Running task selinux... 2024-04-26T21:17:34.515 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-26T21:17:34.516 DEBUG:teuthology.orchestra.run.smithi046:> sudo service auditd rotate 2024-04-26T21:17:34.567 INFO:teuthology.orchestra.run.smithi046.stdout:Rotating logs: 2024-04-26T21:17:34.569 DEBUG:teuthology.orchestra.run.smithi050:> sudo service auditd rotate 2024-04-26T21:17:34.647 INFO:teuthology.orchestra.run.smithi050.stdout:Rotating logs: 2024-04-26T21:17:34.649 DEBUG:teuthology.orchestra.run.smithi142:> sudo service auditd rotate 2024-04-26T21:17:34.718 INFO:teuthology.orchestra.run.smithi142.stdout:Rotating logs: 2024-04-26T21:17:34.720 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-26T21:17:34.720 DEBUG:teuthology.orchestra.run.smithi046:> /usr/sbin/getenforce 2024-04-26T21:17:34.739 INFO:teuthology.orchestra.run.smithi046.stdout:Permissive 2024-04-26T21:17:34.739 DEBUG:teuthology.orchestra.run.smithi050:> /usr/sbin/getenforce 2024-04-26T21:17:34.764 INFO:teuthology.orchestra.run.smithi050.stdout:Permissive 2024-04-26T21:17:34.764 DEBUG:teuthology.orchestra.run.smithi142:> /usr/sbin/getenforce 2024-04-26T21:17:34.791 INFO:teuthology.orchestra.run.smithi142.stdout:Permissive 2024-04-26T21:17:34.791 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi046.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi050.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi142.front.sepia.ceph.com': 'permissive'} 2024-04-26T21:17:34.791 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T21:17:34.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T21:17:34.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T21:17:34.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T21:17:34.844 DEBUG:teuthology.orchestra.run.smithi142:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T21:17:34.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T21:17:34.869 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-26T21:17:34.869 DEBUG:teuthology.orchestra.run.smithi046:> sudo /usr/sbin/setenforce permissive 2024-04-26T21:17:34.894 DEBUG:teuthology.orchestra.run.smithi050:> sudo /usr/sbin/setenforce permissive 2024-04-26T21:17:34.921 DEBUG:teuthology.orchestra.run.smithi142:> sudo /usr/sbin/setenforce permissive 2024-04-26T21:17:34.994 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-26T21:17:35.004 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-26T21:17:35.010 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-26T21:17:35.129 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-26T21:17:35.144 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-26T21:17:35.145 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi046.front.sepia.ceph.com,smithi050.front.sepia.ceph.com,smithi142.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-26T21:21:27.237 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi046.front.sepia.ceph.com'), Remote(name='ubuntu@smithi050.front.sepia.ceph.com'), Remote(name='ubuntu@smithi142.front.sepia.ceph.com')] 2024-04-26T21:21:27.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-04-26T21:21:27.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T21:21:27.314 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-04-26T21:21:27.406 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-04-26T21:21:27.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-04-26T21:21:27.407 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T21:21:27.526 DEBUG:teuthology.orchestra.run.smithi050:> true 2024-04-26T21:21:27.620 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-04-26T21:21:27.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-04-26T21:21:27.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi142.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T21:21:27.704 DEBUG:teuthology.orchestra.run.smithi142:> true 2024-04-26T21:21:27.795 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi142.front.sepia.ceph.com' 2024-04-26T21:21:27.796 INFO:teuthology.run_tasks:Running task clock... 2024-04-26T21:21:27.807 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-26T21:21:27.808 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T21:21:27.808 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T21:21:27.811 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T21:21:27.811 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T21:21:27.813 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T21:21:27.813 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T21:21:27.841 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T21:21:27.845 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T21:21:27.857 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T21:21:27.861 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T21:21:27.866 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T21:21:27.881 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T21:21:27.883 INFO:teuthology.orchestra.run.smithi046.stderr:sudo: ntpd: command not found 2024-04-26T21:21:27.886 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: ntpd: command not found 2024-04-26T21:21:27.894 INFO:teuthology.orchestra.run.smithi046.stdout:506 Cannot talk to daemon 2024-04-26T21:21:27.898 INFO:teuthology.orchestra.run.smithi050.stdout:506 Cannot talk to daemon 2024-04-26T21:21:27.905 INFO:teuthology.orchestra.run.smithi142.stderr:sudo: ntpd: command not found 2024-04-26T21:21:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T21:21:27.912 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T21:21:27.916 INFO:teuthology.orchestra.run.smithi142.stdout:506 Cannot talk to daemon 2024-04-26T21:21:27.922 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T21:21:27.926 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T21:21:27.929 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T21:21:27.943 INFO:teuthology.orchestra.run.smithi142.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T21:21:27.985 INFO:teuthology.orchestra.run.smithi050.stderr:bash: line 1: ntpq: command not found 2024-04-26T21:21:27.986 INFO:teuthology.orchestra.run.smithi046.stderr:bash: line 1: ntpq: command not found 2024-04-26T21:21:27.988 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T21:21:27.988 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2024-04-26T21:21:27.988 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:21:27.988 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:21:27.988 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:21:27.988 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:21:27.990 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T21:21:27.990 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2024-04-26T21:21:27.990 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:21:27.990 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:21:27.990 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:21:27.990 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:21:27.999 INFO:teuthology.orchestra.run.smithi142.stderr:bash: line 1: ntpq: command not found 2024-04-26T21:21:28.003 INFO:teuthology.orchestra.run.smithi142.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T21:21:28.003 INFO:teuthology.orchestra.run.smithi142.stdout:=============================================================================== 2024-04-26T21:21:28.003 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:21:28.003 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:21:28.003 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:21:28.003 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:21:28.003 INFO:teuthology.run_tasks:Running task install... 2024-04-26T21:21:28.016 DEBUG:teuthology.task.install:project ceph 2024-04-26T21:21:28.016 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387'}} 2024-04-26T21:21:28.016 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387'} 2024-04-26T21:21:28.016 INFO:teuthology.task.install:Using flavor: default 2024-04-26T21:21:28.024 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-26T21:21:28.024 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-26T21:21:28.025 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-26T21:21:28.025 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T21:21:28.028 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-26T21:21:28.028 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T21:21:28.029 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-26T21:21:28.029 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T21:21:28.181 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-26T21:21:28.181 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-26T21:21:28.252 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-26T21:21:28.252 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-26T21:21:28.264 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-26T21:21:28.264 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-26T21:21:28.360 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T21:21:28.360 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-04-26T21:21:28.360 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T21:21:28.385 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T21:21:28.386 DEBUG:teuthology.orchestra.run.smithi142:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T21:21:28.443 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T21:21:28.444 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-26T21:21:28.444 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T21:21:28.451 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T21:21:28.451 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T21:21:28.451 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T21:21:28.452 DEBUG:teuthology.orchestra.run.smithi142:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T21:21:28.474 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T21:21:28.474 DEBUG:teuthology.orchestra.run.smithi046:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T21:21:28.479 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T21:21:28.480 DEBUG:teuthology.orchestra.run.smithi050:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T21:21:28.524 DEBUG:teuthology.orchestra.run.smithi142:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T21:21:28.544 DEBUG:teuthology.orchestra.run.smithi046:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T21:21:28.547 DEBUG:teuthology.orchestra.run.smithi050:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T21:21:28.587 INFO:teuthology.orchestra.run.smithi142.stdout:check_obsoletes = 1 2024-04-26T21:21:28.588 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum clean all 2024-04-26T21:21:28.625 DEBUG:teuthology.orchestra.run.smithi050:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T21:21:28.626 DEBUG:teuthology.orchestra.run.smithi046:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T21:21:28.690 INFO:teuthology.orchestra.run.smithi050.stdout:check_obsoletes = 1 2024-04-26T21:21:28.692 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum clean all 2024-04-26T21:21:28.693 INFO:teuthology.orchestra.run.smithi046.stdout:check_obsoletes = 1 2024-04-26T21:21:28.695 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum clean all 2024-04-26T21:21:28.812 INFO:teuthology.orchestra.run.smithi142.stdout:48 files removed 2024-04-26T21:21:28.839 DEBUG:teuthology.orchestra.run.smithi142:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T21:21:28.927 INFO:teuthology.orchestra.run.smithi050.stdout:48 files removed 2024-04-26T21:21:28.938 INFO:teuthology.orchestra.run.smithi046.stdout:48 files removed 2024-04-26T21:21:28.954 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T21:21:28.965 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T21:21:29.571 INFO:teuthology.orchestra.run.smithi142.stdout:ceph packages for x86_64 177 kB/s | 84 kB 00:00 2024-04-26T21:21:29.666 INFO:teuthology.orchestra.run.smithi050.stdout:ceph packages for x86_64 192 kB/s | 84 kB 00:00 2024-04-26T21:21:29.767 INFO:teuthology.orchestra.run.smithi046.stdout:ceph packages for x86_64 158 kB/s | 84 kB 00:00 2024-04-26T21:21:29.988 INFO:teuthology.orchestra.run.smithi142.stdout:ceph noarch packages 31 kB/s | 12 kB 00:00 2024-04-26T21:21:30.074 INFO:teuthology.orchestra.run.smithi050.stdout:ceph noarch packages 32 kB/s | 12 kB 00:00 2024-04-26T21:21:30.159 INFO:teuthology.orchestra.run.smithi046.stdout:ceph noarch packages 34 kB/s | 12 kB 00:00 2024-04-26T21:21:30.430 INFO:teuthology.orchestra.run.smithi142.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-04-26T21:21:30.458 INFO:teuthology.orchestra.run.smithi050.stdout:ceph source packages 5.5 kB/s | 1.9 kB 00:00 2024-04-26T21:21:30.535 INFO:teuthology.orchestra.run.smithi046.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-04-26T21:21:30.980 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-04-26T21:21:31.535 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - BaseOS 8.3 MB/s | 8.0 MB 00:00 2024-04-26T21:21:32.091 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.0 MB 00:01 2024-04-26T21:21:34.201 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-04-26T21:21:37.111 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - AppStream 4.3 MB/s | 19 MB 00:04 2024-04-26T21:21:38.787 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - CRB 6.9 MB/s | 6.2 MB 00:00 2024-04-26T21:21:40.611 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 16 kB 00:00 2024-04-26T21:21:41.680 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - CRB 7.1 MB/s | 6.2 MB 00:00 2024-04-26T21:21:42.125 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-04-26T21:21:43.195 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-04-26T21:21:44.491 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-04-26T21:21:47.365 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-26T21:21:49.566 INFO:teuthology.orchestra.run.smithi050.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T21:21:49.566 INFO:teuthology.orchestra.run.smithi050.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T21:21:49.578 INFO:teuthology.orchestra.run.smithi050.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T21:21:49.578 INFO:teuthology.orchestra.run.smithi050.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T21:21:49.579 INFO:teuthology.orchestra.run.smithi050.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T21:21:49.580 INFO:teuthology.orchestra.run.smithi050.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T21:21:49.582 INFO:teuthology.orchestra.run.smithi050.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T21:21:49.584 INFO:teuthology.orchestra.run.smithi050.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T21:21:49.585 INFO:teuthology.orchestra.run.smithi050.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T21:21:49.649 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-04-26T21:21:49.664 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-04-26T21:21:49.671 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================= 2024-04-26T21:21:49.671 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-04-26T21:21:49.671 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================= 2024-04-26T21:21:49.671 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-04-26T21:21:49.671 INFO:teuthology.orchestra.run.smithi050.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T21:21:49.672 INFO:teuthology.orchestra.run.smithi050.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-26T21:21:49.672 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T21:21:49.672 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-26T21:21:49.672 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-26T21:21:49.672 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-26T21:21:49.672 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-26T21:21:49.672 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-26T21:21:49.672 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-26T21:21:49.672 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-26T21:21:49.672 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-26T21:21:49.672 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-26T21:21:49.672 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-26T21:21:49.672 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-26T21:21:49.673 INFO:teuthology.orchestra.run.smithi050.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-26T21:21:49.673 INFO:teuthology.orchestra.run.smithi050.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-26T21:21:49.673 INFO:teuthology.orchestra.run.smithi050.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-26T21:21:49.673 INFO:teuthology.orchestra.run.smithi050.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T21:21:49.673 INFO:teuthology.orchestra.run.smithi050.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T21:21:49.673 INFO:teuthology.orchestra.run.smithi050.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T21:21:49.673 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-26T21:21:49.673 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-26T21:21:49.673 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-26T21:21:49.673 INFO:teuthology.orchestra.run.smithi050.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T21:21:49.673 INFO:teuthology.orchestra.run.smithi050.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T21:21:49.673 INFO:teuthology.orchestra.run.smithi050.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T21:21:49.673 INFO:teuthology.orchestra.run.smithi050.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T21:21:49.673 INFO:teuthology.orchestra.run.smithi050.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T21:21:49.674 INFO:teuthology.orchestra.run.smithi050.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T21:21:49.674 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-26T21:21:49.674 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-26T21:21:49.674 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-26T21:21:49.674 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-26T21:21:49.674 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-26T21:21:49.674 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T21:21:49.674 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-26T21:21:49.674 INFO:teuthology.orchestra.run.smithi050.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T21:21:49.674 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrading: 2024-04-26T21:21:49.674 INFO:teuthology.orchestra.run.smithi050.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-26T21:21:49.674 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T21:21:49.675 INFO:teuthology.orchestra.run.smithi050.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T21:21:49.675 INFO:teuthology.orchestra.run.smithi050.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T21:21:49.675 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-04-26T21:21:49.675 INFO:teuthology.orchestra.run.smithi050.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T21:21:49.675 INFO:teuthology.orchestra.run.smithi050.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T21:21:49.675 INFO:teuthology.orchestra.run.smithi050.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T21:21:49.675 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-26T21:21:49.675 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-26T21:21:49.675 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-26T21:21:49.675 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-26T21:21:49.675 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-26T21:21:49.675 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-26T21:21:49.675 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-26T21:21:49.676 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-26T21:21:49.676 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T21:21:49.676 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T21:21:49.676 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T21:21:49.676 INFO:teuthology.orchestra.run.smithi050.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T21:21:49.676 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T21:21:49.676 INFO:teuthology.orchestra.run.smithi050.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T21:21:49.676 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T21:21:49.676 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T21:21:49.676 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-26T21:21:49.676 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T21:21:49.676 INFO:teuthology.orchestra.run.smithi050.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T21:21:49.676 INFO:teuthology.orchestra.run.smithi050.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T21:21:49.677 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T21:21:49.677 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T21:21:49.677 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-26T21:21:49.677 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T21:21:49.677 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T21:21:49.677 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T21:21:49.677 INFO:teuthology.orchestra.run.smithi050.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T21:21:49.677 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T21:21:49.677 INFO:teuthology.orchestra.run.smithi050.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T21:21:49.677 INFO:teuthology.orchestra.run.smithi050.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T21:21:49.677 INFO:teuthology.orchestra.run.smithi050.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T21:21:49.677 INFO:teuthology.orchestra.run.smithi050.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T21:21:49.677 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T21:21:49.678 INFO:teuthology.orchestra.run.smithi050.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T21:21:49.678 INFO:teuthology.orchestra.run.smithi050.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T21:21:49.678 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T21:21:49.678 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T21:21:49.678 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T21:21:49.678 INFO:teuthology.orchestra.run.smithi050.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T21:21:49.678 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T21:21:49.678 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T21:21:49.678 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T21:21:49.678 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-26T21:21:49.678 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-26T21:21:49.678 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T21:21:49.678 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T21:21:49.679 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T21:21:49.679 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T21:21:49.679 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T21:21:49.679 INFO:teuthology.orchestra.run.smithi050.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T21:21:49.679 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T21:21:49.679 INFO:teuthology.orchestra.run.smithi050.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T21:21:49.679 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T21:21:49.679 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T21:21:49.679 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T21:21:49.679 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T21:21:49.679 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T21:21:49.679 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T21:21:49.679 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T21:21:49.679 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T21:21:49.680 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T21:21:49.680 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T21:21:49.680 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T21:21:49.680 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T21:21:49.680 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T21:21:49.680 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T21:21:49.680 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T21:21:49.680 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T21:21:49.680 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T21:21:49.680 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T21:21:49.680 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T21:21:49.680 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T21:21:49.680 INFO:teuthology.orchestra.run.smithi050.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T21:21:49.680 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T21:21:49.681 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T21:21:49.681 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T21:21:49.681 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T21:21:49.681 INFO:teuthology.orchestra.run.smithi050.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T21:21:49.681 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T21:21:49.681 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T21:21:49.681 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T21:21:49.681 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T21:21:49.681 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T21:21:49.681 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T21:21:49.681 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T21:21:49.681 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T21:21:49.681 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T21:21:49.682 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T21:21:49.682 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T21:21:49.682 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T21:21:49.682 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T21:21:49.682 INFO:teuthology.orchestra.run.smithi050.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T21:21:49.682 INFO:teuthology.orchestra.run.smithi050.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T21:21:49.682 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T21:21:49.682 INFO:teuthology.orchestra.run.smithi050.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T21:21:49.682 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T21:21:49.682 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T21:21:49.682 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T21:21:49.682 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T21:21:49.682 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T21:21:49.682 INFO:teuthology.orchestra.run.smithi050.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T21:21:49.683 INFO:teuthology.orchestra.run.smithi050.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T21:21:49.683 INFO:teuthology.orchestra.run.smithi050.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T21:21:49.683 INFO:teuthology.orchestra.run.smithi050.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T21:21:49.683 INFO:teuthology.orchestra.run.smithi050.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T21:21:49.683 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T21:21:49.683 INFO:teuthology.orchestra.run.smithi050.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T21:21:49.683 INFO:teuthology.orchestra.run.smithi050.stdout:Installing weak dependencies: 2024-04-26T21:21:49.683 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T21:21:49.683 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T21:21:49.683 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-04-26T21:21:49.683 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================= 2024-04-26T21:21:49.683 INFO:teuthology.orchestra.run.smithi050.stdout:Install 148 Packages 2024-04-26T21:21:49.683 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrade 4 Packages 2024-04-26T21:21:49.683 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T21:21:49.684 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 222 M 2024-04-26T21:21:49.684 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-04-26T21:21:50.308 INFO:teuthology.orchestra.run.smithi050.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 39 kB/s | 6.6 kB 00:00 2024-04-26T21:21:50.533 INFO:teuthology.orchestra.run.smithi050.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 3.9 MB/s | 889 kB 00:00 2024-04-26T21:21:50.583 INFO:teuthology.orchestra.run.smithi050.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 2.8 MB/s | 145 kB 00:00 2024-04-26T21:21:50.908 INFO:teuthology.orchestra.run.smithi050.stdout:(4/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 7.0 MB/s | 5.4 MB 00:00 2024-04-26T21:21:50.958 INFO:teuthology.orchestra.run.smithi050.stdout:(5/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 6.3 MB/s | 2.4 MB 00:00 2024-04-26T21:21:51.092 INFO:teuthology.orchestra.run.smithi050.stdout:(6/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 8.3 MB/s | 1.5 MB 00:00 2024-04-26T21:21:51.517 INFO:teuthology.orchestra.run.smithi050.stdout:(7/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 8.4 MB/s | 4.7 MB 00:00 2024-04-26T21:21:51.528 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - AppStream 1.0 MB/s | 19 MB 00:19 2024-04-26T21:21:51.866 INFO:teuthology.orchestra.run.smithi046.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T21:21:51.867 INFO:teuthology.orchestra.run.smithi046.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T21:21:51.878 INFO:teuthology.orchestra.run.smithi046.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T21:21:51.878 INFO:teuthology.orchestra.run.smithi046.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T21:21:51.879 INFO:teuthology.orchestra.run.smithi046.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T21:21:51.880 INFO:teuthology.orchestra.run.smithi046.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T21:21:51.882 INFO:teuthology.orchestra.run.smithi046.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T21:21:51.884 INFO:teuthology.orchestra.run.smithi046.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T21:21:51.885 INFO:teuthology.orchestra.run.smithi046.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T21:21:51.949 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-04-26T21:21:51.958 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================= 2024-04-26T21:21:51.958 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-04-26T21:21:51.959 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================= 2024-04-26T21:21:51.959 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-04-26T21:21:51.959 INFO:teuthology.orchestra.run.smithi046.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T21:21:51.959 INFO:teuthology.orchestra.run.smithi046.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-26T21:21:51.959 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T21:21:51.959 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-26T21:21:51.959 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-26T21:21:51.959 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-26T21:21:51.959 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-26T21:21:51.959 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-26T21:21:51.959 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-26T21:21:51.959 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-26T21:21:51.959 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-26T21:21:51.959 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-26T21:21:51.960 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-26T21:21:51.960 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-26T21:21:51.960 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-26T21:21:51.960 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-26T21:21:51.960 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-26T21:21:51.960 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T21:21:51.960 INFO:teuthology.orchestra.run.smithi046.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T21:21:51.960 INFO:teuthology.orchestra.run.smithi046.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T21:21:51.960 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-26T21:21:51.960 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-26T21:21:51.960 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-26T21:21:51.960 INFO:teuthology.orchestra.run.smithi046.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T21:21:51.960 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T21:21:51.961 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T21:21:51.961 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T21:21:51.961 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T21:21:51.961 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T21:21:51.961 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-26T21:21:51.961 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-26T21:21:51.961 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-26T21:21:51.961 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-26T21:21:51.961 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-26T21:21:51.961 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T21:21:51.961 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-26T21:21:51.961 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T21:21:51.962 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrading: 2024-04-26T21:21:51.962 INFO:teuthology.orchestra.run.smithi046.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-26T21:21:51.962 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T21:21:51.962 INFO:teuthology.orchestra.run.smithi046.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T21:21:51.962 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T21:21:51.962 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-04-26T21:21:51.962 INFO:teuthology.orchestra.run.smithi046.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T21:21:51.962 INFO:teuthology.orchestra.run.smithi046.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T21:21:51.962 INFO:teuthology.orchestra.run.smithi046.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T21:21:51.962 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-26T21:21:51.962 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-26T21:21:51.962 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-26T21:21:51.962 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-26T21:21:51.963 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-26T21:21:51.963 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-26T21:21:51.963 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-26T21:21:51.963 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-26T21:21:51.963 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T21:21:51.964 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T21:21:51.964 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T21:21:51.964 INFO:teuthology.orchestra.run.smithi046.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T21:21:51.964 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T21:21:51.964 INFO:teuthology.orchestra.run.smithi046.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T21:21:51.964 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T21:21:51.964 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T21:21:51.964 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-26T21:21:51.964 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T21:21:51.964 INFO:teuthology.orchestra.run.smithi046.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T21:21:51.964 INFO:teuthology.orchestra.run.smithi046.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T21:21:51.964 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T21:21:51.964 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T21:21:51.964 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-26T21:21:51.965 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T21:21:51.965 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T21:21:51.965 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T21:21:51.965 INFO:teuthology.orchestra.run.smithi046.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T21:21:51.965 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T21:21:51.965 INFO:teuthology.orchestra.run.smithi046.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T21:21:51.965 INFO:teuthology.orchestra.run.smithi046.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T21:21:51.965 INFO:teuthology.orchestra.run.smithi046.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T21:21:51.965 INFO:teuthology.orchestra.run.smithi046.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T21:21:51.965 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T21:21:51.965 INFO:teuthology.orchestra.run.smithi046.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T21:21:51.965 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T21:21:51.966 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T21:21:51.966 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T21:21:51.966 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T21:21:51.966 INFO:teuthology.orchestra.run.smithi046.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T21:21:51.966 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T21:21:51.966 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T21:21:51.966 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T21:21:51.966 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-26T21:21:51.966 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-26T21:21:51.966 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T21:21:51.967 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T21:21:51.967 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T21:21:51.967 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T21:21:51.967 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T21:21:51.967 INFO:teuthology.orchestra.run.smithi046.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T21:21:51.967 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T21:21:51.967 INFO:teuthology.orchestra.run.smithi046.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T21:21:51.967 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T21:21:51.967 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T21:21:51.967 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T21:21:51.968 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T21:21:51.968 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T21:21:51.968 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T21:21:51.968 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T21:21:51.968 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T21:21:51.968 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T21:21:51.968 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T21:21:51.968 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T21:21:51.968 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T21:21:51.968 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T21:21:51.969 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T21:21:51.969 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T21:21:51.969 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T21:21:51.969 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T21:21:51.969 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T21:21:51.969 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T21:21:51.970 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T21:21:51.970 INFO:teuthology.orchestra.run.smithi046.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T21:21:51.970 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T21:21:51.970 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T21:21:51.970 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T21:21:51.970 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T21:21:51.970 INFO:teuthology.orchestra.run.smithi046.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T21:21:51.970 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T21:21:51.970 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T21:21:51.970 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T21:21:51.971 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T21:21:51.971 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T21:21:51.971 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T21:21:51.971 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T21:21:51.971 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T21:21:51.972 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T21:21:51.972 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T21:21:51.972 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T21:21:51.972 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T21:21:51.972 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T21:21:51.972 INFO:teuthology.orchestra.run.smithi046.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T21:21:51.972 INFO:teuthology.orchestra.run.smithi046.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T21:21:51.972 INFO:teuthology.orchestra.run.smithi046.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T21:21:51.972 INFO:teuthology.orchestra.run.smithi046.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T21:21:51.972 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T21:21:51.973 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T21:21:51.973 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T21:21:51.973 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T21:21:51.973 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T21:21:51.973 INFO:teuthology.orchestra.run.smithi046.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T21:21:51.973 INFO:teuthology.orchestra.run.smithi046.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T21:21:51.974 INFO:teuthology.orchestra.run.smithi046.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T21:21:51.974 INFO:teuthology.orchestra.run.smithi046.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T21:21:51.974 INFO:teuthology.orchestra.run.smithi046.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T21:21:51.974 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T21:21:51.974 INFO:teuthology.orchestra.run.smithi046.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T21:21:51.974 INFO:teuthology.orchestra.run.smithi046.stdout:Installing weak dependencies: 2024-04-26T21:21:51.974 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T21:21:51.974 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:21:51.974 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-04-26T21:21:51.974 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================= 2024-04-26T21:21:51.975 INFO:teuthology.orchestra.run.smithi046.stdout:Install 148 Packages 2024-04-26T21:21:51.975 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrade 4 Packages 2024-04-26T21:21:51.975 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:21:51.976 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 222 M 2024-04-26T21:21:51.976 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-04-26T21:21:52.542 INFO:teuthology.orchestra.run.smithi050.stdout:(8/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x86 12 MB/s | 17 MB 00:01 2024-04-26T21:21:52.593 INFO:teuthology.orchestra.run.smithi050.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 504 kB/s | 25 kB 00:00 2024-04-26T21:21:52.735 INFO:teuthology.orchestra.run.smithi050.stdout:(10/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el 8.7 MB/s | 11 MB 00:01 2024-04-26T21:21:52.818 INFO:teuthology.orchestra.run.smithi050.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 2.6 MB/s | 225 kB 00:00 2024-04-26T21:21:52.865 INFO:teuthology.orchestra.run.smithi046.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 25 kB/s | 6.6 kB 00:00 2024-04-26T21:21:52.885 INFO:teuthology.orchestra.run.smithi050.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 406 kB/s | 27 kB 00:00 2024-04-26T21:21:52.944 INFO:teuthology.orchestra.run.smithi050.stdout:(13/152): libcephfs-devel-19.0.0-3306.gbc612ffb 542 kB/s | 32 kB 00:00 2024-04-26T21:21:53.103 INFO:teuthology.orchestra.run.smithi050.stdout:(14/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 4.6 MB/s | 747 kB 00:00 2024-04-26T21:21:53.186 INFO:teuthology.orchestra.run.smithi050.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 1.9 MB/s | 163 kB 00:00 2024-04-26T21:21:53.270 INFO:teuthology.orchestra.run.smithi050.stdout:(16/152): librados-devel-19.0.0-3306.gbc612ffb. 1.5 MB/s | 127 kB 00:00 2024-04-26T21:21:53.362 INFO:teuthology.orchestra.run.smithi050.stdout:(17/152): libradosstriper1-19.0.0-3306.gbc612ff 5.3 MB/s | 501 kB 00:00 2024-04-26T21:21:53.663 INFO:teuthology.orchestra.run.smithi050.stdout:(18/152): ceph-common-19.0.0-3306.gbc612ffb.el9 6.0 MB/s | 21 MB 00:03 2024-04-26T21:21:53.748 INFO:teuthology.orchestra.run.smithi046.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.0 MB/s | 889 kB 00:00 2024-04-26T21:21:53.763 INFO:teuthology.orchestra.run.smithi050.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 452 kB/s | 45 kB 00:00 2024-04-26T21:21:53.871 INFO:teuthology.orchestra.run.smithi050.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 1.2 MB/s | 135 kB 00:00 2024-04-26T21:21:53.907 INFO:teuthology.orchestra.run.smithi046.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 918 kB/s | 145 kB 00:00 2024-04-26T21:21:54.030 INFO:teuthology.orchestra.run.smithi050.stdout:(21/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 8.0 MB/s | 5.4 MB 00:00 2024-04-26T21:21:54.055 INFO:teuthology.orchestra.run.smithi050.stdout:(22/152): python3-cephfs-19.0.0-3306.gbc612ffb. 880 kB/s | 161 kB 00:00 2024-04-26T21:21:54.106 INFO:teuthology.orchestra.run.smithi050.stdout:(23/152): python3-rados-19.0.0-3306.gbc612ffb.e 4.2 MB/s | 321 kB 00:00 2024-04-26T21:21:54.156 INFO:teuthology.orchestra.run.smithi050.stdout:(24/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 2.9 MB/s | 298 kB 00:00 2024-04-26T21:21:54.190 INFO:teuthology.orchestra.run.smithi050.stdout:(25/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 1.2 MB/s | 99 kB 00:00 2024-04-26T21:21:54.248 INFO:teuthology.orchestra.run.smithi050.stdout:(26/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 926 kB/s | 85 kB 00:00 2024-04-26T21:21:54.365 INFO:teuthology.orchestra.run.smithi050.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 1.4 MB/s | 171 kB 00:00 2024-04-26T21:21:54.474 INFO:teuthology.orchestra.run.smithi050.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3306.g 315 kB/s | 34 kB 00:00 2024-04-26T21:21:54.591 INFO:teuthology.orchestra.run.smithi050.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 1.2 MB/s | 142 kB 00:00 2024-04-26T21:21:54.641 INFO:teuthology.orchestra.run.smithi050.stdout:(30/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 6.9 MB/s | 3.1 MB 00:00 2024-04-26T21:21:55.007 INFO:teuthology.orchestra.run.smithi046.stdout:(4/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 2.1 MB/s | 2.4 MB 00:01 2024-04-26T21:21:55.316 INFO:teuthology.orchestra.run.smithi050.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 4.9 MB/s | 3.5 MB 00:00 2024-04-26T21:21:55.425 INFO:teuthology.orchestra.run.smithi050.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 2.2 MB/s | 249 kB 00:00 2024-04-26T21:21:55.507 INFO:teuthology.orchestra.run.smithi046.stdout:(5/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 3.0 MB/s | 1.5 MB 00:00 2024-04-26T21:21:55.525 INFO:teuthology.orchestra.run.smithi050.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 494 kB/s | 49 kB 00:00 2024-04-26T21:21:55.634 INFO:teuthology.orchestra.run.smithi050.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3306.gb 152 kB/s | 16 kB 00:00 2024-04-26T21:21:55.699 INFO:teuthology.orchestra.run.smithi046.stdout:(6/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 1.7 MB/s | 5.4 MB 00:03 2024-04-26T21:21:55.734 INFO:teuthology.orchestra.run.smithi050.stdout:(35/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 2.7 MB/s | 273 kB 00:00 2024-04-26T21:21:55.843 INFO:teuthology.orchestra.run.smithi050.stdout:(36/152): ceph-mgr-diskprediction-local-19.0.0- 6.2 MB/s | 7.4 MB 00:01 2024-04-26T21:21:55.926 INFO:teuthology.orchestra.run.smithi050.stdout:(37/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 219 kB/s | 18 kB 00:00 2024-04-26T21:21:55.951 INFO:teuthology.orchestra.run.smithi050.stdout:(38/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 3.4 MB/s | 762 kB 00:00 2024-04-26T21:21:56.077 INFO:teuthology.orchestra.run.smithi050.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 535 kB/s | 80 kB 00:00 2024-04-26T21:21:56.307 INFO:teuthology.orchestra.run.smithi046.stdout:(7/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 5.9 MB/s | 4.7 MB 00:00 2024-04-26T21:21:56.577 INFO:teuthology.orchestra.run.smithi050.stdout:(40/152): ceph-test-19.0.0-3306.gbc612ffb.el9.x 12 MB/s | 49 MB 00:03 2024-04-26T21:21:56.603 INFO:teuthology.orchestra.run.smithi050.stdout:(41/152): libgfortran-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 803 kB 00:00 2024-04-26T21:21:56.629 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.2 MB 00:01 2024-04-26T21:21:56.636 INFO:teuthology.orchestra.run.smithi050.stdout:(42/152): libquadmath-11.4.1-3.el9.x86_64.rpm 349 kB/s | 195 kB 00:00 2024-04-26T21:21:56.661 INFO:teuthology.orchestra.run.smithi050.stdout:(43/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.0 MB/s | 253 kB 00:00 2024-04-26T21:21:56.712 INFO:teuthology.orchestra.run.smithi050.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-26T21:21:56.745 INFO:teuthology.orchestra.run.smithi050.stdout:(45/152): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.2 MB 00:00 2024-04-26T21:21:56.770 INFO:teuthology.orchestra.run.smithi050.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-26T21:21:56.796 INFO:teuthology.orchestra.run.smithi050.stdout:(47/152): python3-ply-3.11-14.el9.noarch.rpm 668 kB/s | 106 kB 00:00 2024-04-26T21:21:56.821 INFO:teuthology.orchestra.run.smithi050.stdout:(48/152): python3-urllib3-1.26.5-5.el9.noarch.r 2.8 MB/s | 215 kB 00:00 2024-04-26T21:21:56.963 INFO:teuthology.orchestra.run.smithi050.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 755 kB/s | 107 kB 00:00 2024-04-26T21:21:57.005 INFO:teuthology.orchestra.run.smithi050.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 516 kB/s | 21 kB 00:00 2024-04-26T21:21:57.155 INFO:teuthology.orchestra.run.smithi050.stdout:(51/152): automake-1.16.2-8.el9.noarch.rpm 1.8 MB/s | 689 kB 00:00 2024-04-26T21:21:57.181 INFO:teuthology.orchestra.run.smithi050.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 1.7 MB/s | 313 kB 00:00 2024-04-26T21:21:57.214 INFO:teuthology.orchestra.run.smithi050.stdout:(53/152): bison-3.7.4-5.el9.x86_64.rpm 2.2 MB/s | 943 kB 00:00 2024-04-26T21:21:57.233 INFO:teuthology.orchestra.run.smithi046.stdout:(8/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el9 11 MB/s | 11 MB 00:00 2024-04-26T21:21:57.239 INFO:teuthology.orchestra.run.smithi050.stdout:(54/152): flexiblas-3.0.4-8.el9.x86_64.rpm 390 kB/s | 33 kB 00:00 2024-04-26T21:21:57.265 INFO:teuthology.orchestra.run.smithi050.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 363 kB/s | 18 kB 00:00 2024-04-26T21:21:57.290 INFO:teuthology.orchestra.run.smithi050.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-04-26T21:21:57.340 INFO:teuthology.orchestra.run.smithi050.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 902 kB/s | 45 kB 00:00 2024-04-26T21:21:57.365 INFO:teuthology.orchestra.run.smithi050.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 160 kB 00:00 2024-04-26T21:21:57.433 INFO:teuthology.orchestra.run.smithi046.stdout:(9/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x86 9.7 MB/s | 17 MB 00:01 2024-04-26T21:21:57.458 INFO:teuthology.orchestra.run.smithi046.stdout:(10/152): ceph-selinux-19.0.0-3306.gbc612ffb.el 112 kB/s | 25 kB 00:00 2024-04-26T21:21:57.534 INFO:teuthology.orchestra.run.smithi046.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 2.9 MB/s | 225 kB 00:00 2024-04-26T21:21:57.566 INFO:teuthology.orchestra.run.smithi050.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 2.9 MB/s | 662 kB 00:00 2024-04-26T21:21:57.609 INFO:teuthology.orchestra.run.smithi046.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 361 kB/s | 27 kB 00:00 2024-04-26T21:21:57.616 INFO:teuthology.orchestra.run.smithi050.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 509 kB/s | 25 kB 00:00 2024-04-26T21:21:57.683 INFO:teuthology.orchestra.run.smithi050.stdout:(61/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.0 MB/s | 3.0 MB 00:00 2024-04-26T21:21:57.692 INFO:teuthology.orchestra.run.smithi046.stdout:(13/152): libcephfs-devel-19.0.0-3306.gbc612ffb 381 kB/s | 32 kB 00:00 2024-04-26T21:21:57.708 INFO:teuthology.orchestra.run.smithi050.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 517 kB/s | 47 kB 00:00 2024-04-26T21:21:57.734 INFO:teuthology.orchestra.run.smithi050.stdout:(63/152): libtool-2.4.6-45.el9.x86_64.rpm 1.5 MB/s | 582 kB 00:00 2024-04-26T21:21:57.767 INFO:teuthology.orchestra.run.smithi050.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-04-26T21:21:57.793 INFO:teuthology.orchestra.run.smithi050.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-04-26T21:21:57.801 INFO:teuthology.orchestra.run.smithi046.stdout:(14/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 6.7 MB/s | 747 kB 00:00 2024-04-26T21:21:57.818 INFO:teuthology.orchestra.run.smithi050.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 420 kB/s | 35 kB 00:00 2024-04-26T21:21:57.893 INFO:teuthology.orchestra.run.smithi046.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 1.7 MB/s | 163 kB 00:00 2024-04-26T21:21:57.985 INFO:teuthology.orchestra.run.smithi046.stdout:(16/152): librados-devel-19.0.0-3306.gbc612ffb. 1.4 MB/s | 127 kB 00:00 2024-04-26T21:21:57.987 INFO:teuthology.orchestra.run.smithi142.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 16 kB 00:00 2024-04-26T21:21:58.093 INFO:teuthology.orchestra.run.smithi046.stdout:(17/152): libradosstriper1-19.0.0-3306.gbc612ff 4.5 MB/s | 501 kB 00:00 2024-04-26T21:21:58.318 INFO:teuthology.orchestra.run.smithi050.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 3.2 MB/s | 1.6 MB 00:00 2024-04-26T21:21:58.577 INFO:teuthology.orchestra.run.smithi050.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 3.4 MB/s | 911 kB 00:00 2024-04-26T21:21:58.660 INFO:teuthology.orchestra.run.smithi050.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-04-26T21:21:58.727 INFO:teuthology.orchestra.run.smithi050.stdout:(70/152): openblas-openmp-0.3.21-2.el9.x86_64.r 4.9 MB/s | 4.7 MB 00:00 2024-04-26T21:21:58.735 INFO:teuthology.orchestra.run.smithi046.stdout:(18/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 8.4 MB/s | 5.4 MB 00:00 2024-04-26T21:21:58.819 INFO:teuthology.orchestra.run.smithi046.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 542 kB/s | 45 kB 00:00 2024-04-26T21:21:58.903 INFO:teuthology.orchestra.run.smithi046.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 1.6 MB/s | 135 kB 00:00 2024-04-26T21:21:58.986 INFO:teuthology.orchestra.run.smithi046.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 1.9 MB/s | 161 kB 00:00 2024-04-26T21:21:59.019 INFO:teuthology.orchestra.run.smithi050.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 3.5 MB/s | 1.0 MB 00:00 2024-04-26T21:21:59.078 INFO:teuthology.orchestra.run.smithi050.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-26T21:21:59.078 INFO:teuthology.orchestra.run.smithi046.stdout:(22/152): python3-rados-19.0.0-3306.gbc612ffb.e 3.4 MB/s | 321 kB 00:00 2024-04-26T21:21:59.170 INFO:teuthology.orchestra.run.smithi046.stdout:(23/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 3.2 MB/s | 298 kB 00:00 2024-04-26T21:21:59.228 INFO:teuthology.orchestra.run.smithi050.stdout:(73/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 3.1 MB/s | 4.4 MB 00:01 2024-04-26T21:21:59.254 INFO:teuthology.orchestra.run.smithi046.stdout:(24/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 1.2 MB/s | 99 kB 00:00 2024-04-26T21:21:59.337 INFO:teuthology.orchestra.run.smithi050.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 2.2 MB/s | 244 kB 00:00 2024-04-26T21:21:59.337 INFO:teuthology.orchestra.run.smithi046.stdout:(25/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.0 MB/s | 85 kB 00:00 2024-04-26T21:21:59.454 INFO:teuthology.orchestra.run.smithi050.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.1 MB/s | 248 kB 00:00 2024-04-26T21:21:59.520 INFO:teuthology.orchestra.run.smithi050.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 385 kB/s | 26 kB 00:00 2024-04-26T21:21:59.612 INFO:teuthology.orchestra.run.smithi050.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 209 kB/s | 19 kB 00:00 2024-04-26T21:21:59.721 INFO:teuthology.orchestra.run.smithi050.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-04-26T21:21:59.749 INFO:teuthology.orchestra.run.smithi142.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-26T21:21:59.754 INFO:teuthology.orchestra.run.smithi046.stdout:(26/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 7.4 MB/s | 3.1 MB 00:00 2024-04-26T21:21:59.780 INFO:teuthology.orchestra.run.smithi050.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 598 kB/s | 35 kB 00:00 2024-04-26T21:22:00.038 INFO:teuthology.orchestra.run.smithi050.stdout:(80/152): postgresql-server-13.14-1.el9.x86_64. 4.2 MB/s | 5.8 MB 00:01 2024-04-26T21:22:00.122 INFO:teuthology.orchestra.run.smithi050.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.9 MB/s | 250 kB 00:00 2024-04-26T21:22:00.181 INFO:teuthology.orchestra.run.smithi050.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-04-26T21:22:00.205 INFO:teuthology.orchestra.run.smithi046.stdout:(27/152): ceph-test-19.0.0-3306.gbc612ffb.el9.x 18 MB/s | 49 MB 00:02 2024-04-26T21:22:00.230 INFO:teuthology.orchestra.run.smithi046.stdout:(28/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 359 kB/s | 171 kB 00:00 2024-04-26T21:22:00.231 INFO:teuthology.orchestra.run.smithi050.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-26T21:22:00.264 INFO:teuthology.orchestra.run.smithi046.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3306.g 588 kB/s | 34 kB 00:00 2024-04-26T21:22:00.273 INFO:teuthology.orchestra.run.smithi050.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-04-26T21:22:00.289 INFO:teuthology.orchestra.run.smithi046.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 2.4 MB/s | 142 kB 00:00 2024-04-26T21:22:00.365 INFO:teuthology.orchestra.run.smithi050.stdout:(85/152): python3-protobuf-3.14.0-13.el9.noarch 2.9 MB/s | 269 kB 00:00 2024-04-26T21:22:00.423 INFO:teuthology.orchestra.run.smithi050.stdout:(86/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-04-26T21:22:00.516 INFO:teuthology.orchestra.run.smithi050.stdout:(87/152): python3-pyasn1-modules-0.4.8-6.el9.no 3.0 MB/s | 279 kB 00:00 2024-04-26T21:22:00.531 INFO:teuthology.orchestra.run.smithi046.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 13 MB/s | 3.5 MB 00:00 2024-04-26T21:22:00.566 INFO:teuthology.orchestra.run.smithi050.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-26T21:22:00.615 INFO:teuthology.orchestra.run.smithi046.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 2.9 MB/s | 249 kB 00:00 2024-04-26T21:22:00.616 INFO:teuthology.orchestra.run.smithi050.stdout:(89/152): python3-requests-oauthlib-1.3.0-12.el 1.1 MB/s | 54 kB 00:00 2024-04-26T21:22:00.698 INFO:teuthology.orchestra.run.smithi046.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 593 kB/s | 49 kB 00:00 2024-04-26T21:22:00.765 INFO:teuthology.orchestra.run.smithi046.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3306.gb 247 kB/s | 16 kB 00:00 2024-04-26T21:22:00.840 INFO:teuthology.orchestra.run.smithi046.stdout:(35/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 3.6 MB/s | 273 kB 00:00 2024-04-26T21:22:00.908 INFO:teuthology.orchestra.run.smithi050.stdout:(90/152): python3-babel-2.9.1-2.el9.noarch.rpm 3.3 MB/s | 6.0 MB 00:01 2024-04-26T21:22:00.932 INFO:teuthology.orchestra.run.smithi046.stdout:(36/152): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2024-04-26T21:22:00.949 INFO:teuthology.orchestra.run.smithi050.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-04-26T21:22:00.966 INFO:teuthology.orchestra.run.smithi046.stdout:(37/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 5.9 MB/s | 762 kB 00:00 2024-04-26T21:22:00.983 INFO:teuthology.orchestra.run.smithi046.stdout:(38/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 365 kB/s | 18 kB 00:00 2024-04-26T21:22:01.000 INFO:teuthology.orchestra.run.smithi050.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 882 kB/s | 44 kB 00:00 2024-04-26T21:22:01.091 INFO:teuthology.orchestra.run.smithi046.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 642 kB/s | 80 kB 00:00 2024-04-26T21:22:01.109 INFO:teuthology.orchestra.run.smithi050.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-04-26T21:22:01.158 INFO:teuthology.orchestra.run.smithi046.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2024-04-26T21:22:01.159 INFO:teuthology.orchestra.run.smithi050.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 719 kB/s | 36 kB 00:00 2024-04-26T21:22:01.192 INFO:teuthology.orchestra.run.smithi046.stdout:(41/152): libgfortran-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 803 kB 00:00 2024-04-26T21:22:01.217 INFO:teuthology.orchestra.run.smithi046.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.2 MB/s | 253 kB 00:00 2024-04-26T21:22:01.226 INFO:teuthology.orchestra.run.smithi050.stdout:(95/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.5 MB/s | 5.1 MB 00:01 2024-04-26T21:22:01.243 INFO:teuthology.orchestra.run.smithi050.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 567 kB/s | 47 kB 00:00 2024-04-26T21:22:01.267 INFO:teuthology.orchestra.run.smithi046.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 16 MB/s | 1.2 MB 00:00 2024-04-26T21:22:01.293 INFO:teuthology.orchestra.run.smithi046.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-04-26T21:22:01.293 INFO:teuthology.orchestra.run.smithi050.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-26T21:22:01.318 INFO:teuthology.orchestra.run.smithi046.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-26T21:22:01.343 INFO:teuthology.orchestra.run.smithi046.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-26T21:22:01.351 INFO:teuthology.orchestra.run.smithi050.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 2.4 MB/s | 305 kB 00:00 2024-04-26T21:22:01.368 INFO:teuthology.orchestra.run.smithi046.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-26T21:22:01.402 INFO:teuthology.orchestra.run.smithi050.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 865 kB/s | 94 kB 00:00 2024-04-26T21:22:01.410 INFO:teuthology.orchestra.run.smithi046.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 10 MB/s | 689 kB 00:00 2024-04-26T21:22:01.419 INFO:teuthology.orchestra.run.smithi050.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 952 kB/s | 64 kB 00:00 2024-04-26T21:22:01.444 INFO:teuthology.orchestra.run.smithi046.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 12 MB/s | 943 kB 00:00 2024-04-26T21:22:01.444 INFO:teuthology.orchestra.run.smithi050.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.0 MB/s | 45 kB 00:00 2024-04-26T21:22:01.469 INFO:teuthology.orchestra.run.smithi046.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-04-26T21:22:01.494 INFO:teuthology.orchestra.run.smithi046.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 429 kB/s | 21 kB 00:00 2024-04-26T21:22:01.520 INFO:teuthology.orchestra.run.smithi046.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-04-26T21:22:01.554 INFO:teuthology.orchestra.run.smithi046.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 556 kB/s | 33 kB 00:00 2024-04-26T21:22:01.613 INFO:teuthology.orchestra.run.smithi046.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 313 kB/s | 18 kB 00:00 2024-04-26T21:22:01.671 INFO:teuthology.orchestra.run.smithi046.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 20 MB/s | 3.0 MB 00:00 2024-04-26T21:22:01.688 INFO:teuthology.orchestra.run.smithi046.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 148 kB/s | 11 kB 00:00 2024-04-26T21:22:01.714 INFO:teuthology.orchestra.run.smithi046.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 160 kB 00:00 2024-04-26T21:22:01.730 INFO:teuthology.orchestra.run.smithi046.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-04-26T21:22:01.764 INFO:teuthology.orchestra.run.smithi046.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 662 kB 00:00 2024-04-26T21:22:01.797 INFO:teuthology.orchestra.run.smithi046.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 582 kB 00:00 2024-04-26T21:22:01.823 INFO:teuthology.orchestra.run.smithi046.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 435 kB/s | 25 kB 00:00 2024-04-26T21:22:01.848 INFO:teuthology.orchestra.run.smithi046.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 943 kB/s | 47 kB 00:00 2024-04-26T21:22:01.936 INFO:teuthology.orchestra.run.smithi050.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 43 kB/s | 22 kB 00:00 2024-04-26T21:22:02.057 INFO:teuthology.orchestra.run.smithi046.stdout:(63/152): ceph-common-19.0.0-3306.gbc612ffb.el9 2.2 MB/s | 21 MB 00:09 2024-04-26T21:22:02.082 INFO:teuthology.orchestra.run.smithi046.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 807 kB/s | 188 kB 00:00 2024-04-26T21:22:02.107 INFO:teuthology.orchestra.run.smithi046.stdout:(65/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.0 MB/s | 292 kB 00:00 2024-04-26T21:22:02.149 INFO:teuthology.orchestra.run.smithi046.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 383 kB/s | 35 kB 00:00 2024-04-26T21:22:02.266 INFO:teuthology.orchestra.run.smithi046.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 26 MB/s | 4.7 MB 00:00 2024-04-26T21:22:02.324 INFO:teuthology.orchestra.run.smithi046.stdout:(68/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-04-26T21:22:02.358 INFO:teuthology.orchestra.run.smithi046.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 9.7 MB/s | 911 kB 00:00 2024-04-26T21:22:02.383 INFO:teuthology.orchestra.run.smithi046.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-04-26T21:22:02.434 INFO:teuthology.orchestra.run.smithi046.stdout:(71/152): postgresql-13.14-1.el9.x86_64.rpm 5.6 MB/s | 1.6 MB 00:00 2024-04-26T21:22:02.476 INFO:teuthology.orchestra.run.smithi046.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 11 MB/s | 1.0 MB 00:00 2024-04-26T21:22:02.501 INFO:teuthology.orchestra.run.smithi046.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-04-26T21:22:02.576 INFO:teuthology.orchestra.run.smithi046.stdout:(74/152): postgresql-server-13.14-1.el9.x86_64. 27 MB/s | 5.8 MB 00:00 2024-04-26T21:22:02.601 INFO:teuthology.orchestra.run.smithi046.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 2.4 MB/s | 244 kB 00:00 2024-04-26T21:22:02.626 INFO:teuthology.orchestra.run.smithi046.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-04-26T21:22:02.651 INFO:teuthology.orchestra.run.smithi046.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 514 kB/s | 26 kB 00:00 2024-04-26T21:22:02.677 INFO:teuthology.orchestra.run.smithi046.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 381 kB/s | 19 kB 00:00 2024-04-26T21:22:02.815 INFO:teuthology.orchestra.run.smithi050.stdout:(103/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 7.6 MB/s | 16 MB 00:02 2024-04-26T21:22:02.815 INFO:teuthology.orchestra.run.smithi046.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-04-26T21:22:02.815 INFO:teuthology.orchestra.run.smithi046.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 695 kB/s | 35 kB 00:00 2024-04-26T21:22:02.820 INFO:teuthology.orchestra.run.smithi046.stdout:(81/152): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-04-26T21:22:02.845 INFO:teuthology.orchestra.run.smithi046.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.1 MB/s | 250 kB 00:00 2024-04-26T21:22:02.879 INFO:teuthology.orchestra.run.smithi046.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-04-26T21:22:02.904 INFO:teuthology.orchestra.run.smithi046.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-04-26T21:22:02.929 INFO:teuthology.orchestra.run.smithi046.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 842 kB/s | 42 kB 00:00 2024-04-26T21:22:02.937 INFO:teuthology.orchestra.run.smithi050.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 578 kB/s | 863 kB 00:01 2024-04-26T21:22:02.954 INFO:teuthology.orchestra.run.smithi046.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-04-26T21:22:02.980 INFO:teuthology.orchestra.run.smithi046.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-26T21:22:03.005 INFO:teuthology.orchestra.run.smithi046.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-04-26T21:22:03.031 INFO:teuthology.orchestra.run.smithi046.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-26T21:22:03.055 INFO:teuthology.orchestra.run.smithi046.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-26T21:22:03.139 INFO:teuthology.orchestra.run.smithi046.stdout:(91/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-04-26T21:22:03.164 INFO:teuthology.orchestra.run.smithi046.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 383 kB/s | 42 kB 00:00 2024-04-26T21:22:03.189 INFO:teuthology.orchestra.run.smithi046.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 884 kB/s | 44 kB 00:00 2024-04-26T21:22:03.215 INFO:teuthology.orchestra.run.smithi046.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-26T21:22:03.237 INFO:teuthology.orchestra.run.smithi050.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 421 kB/s | 548 kB 00:01 2024-04-26T21:22:03.240 INFO:teuthology.orchestra.run.smithi046.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 714 kB/s | 36 kB 00:00 2024-04-26T21:22:03.265 INFO:teuthology.orchestra.run.smithi046.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-04-26T21:22:03.291 INFO:teuthology.orchestra.run.smithi046.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 305 kB 00:00 2024-04-26T21:22:03.421 INFO:teuthology.orchestra.run.smithi050.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 40 kB/s | 19 kB 00:00 2024-04-26T21:22:03.432 INFO:teuthology.orchestra.run.smithi046.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 346 kB/s | 58 kB 00:00 2024-04-26T21:22:03.450 INFO:teuthology.orchestra.run.smithi046.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 591 kB/s | 94 kB 00:00 2024-04-26T21:22:03.475 INFO:teuthology.orchestra.run.smithi046.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-04-26T21:22:03.492 INFO:teuthology.orchestra.run.smithi046.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.0 MB/s | 45 kB 00:00 2024-04-26T21:22:03.521 INFO:teuthology.orchestra.run.smithi050.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-04-26T21:22:03.605 INFO:teuthology.orchestra.run.smithi050.stdout:(108/152): gperftools-libs-2.9.1-2.el9.x86_64.r 360 kB/s | 309 kB 00:00 2024-04-26T21:22:03.705 INFO:teuthology.orchestra.run.smithi050.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 265 kB/s | 49 kB 00:00 2024-04-26T21:22:03.775 INFO:teuthology.orchestra.run.smithi046.stdout:(102/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-04-26T21:22:03.805 INFO:teuthology.orchestra.run.smithi050.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 336 kB/s | 67 kB 00:00 2024-04-26T21:22:03.964 INFO:teuthology.orchestra.run.smithi050.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 584 kB/s | 151 kB 00:00 2024-04-26T21:22:04.150 INFO:teuthology.orchestra.run.smithi046.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 33 kB/s | 22 kB 00:00 2024-04-26T21:22:04.426 INFO:teuthology.orchestra.run.smithi046.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 844 kB/s | 548 kB 00:00 2024-04-26T21:22:04.518 INFO:teuthology.orchestra.run.smithi046.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 212 kB/s | 19 kB 00:00 2024-04-26T21:22:04.710 INFO:teuthology.orchestra.run.smithi046.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 553 kB/s | 309 kB 00:00 2024-04-26T21:22:04.714 INFO:teuthology.orchestra.run.smithi050.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 731 kB/s | 548 kB 00:00 2024-04-26T21:22:04.810 INFO:teuthology.orchestra.run.smithi046.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-04-26T21:22:04.823 INFO:teuthology.orchestra.run.smithi050.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 401 kB/s | 43 kB 00:00 2024-04-26T21:22:04.857 INFO:teuthology.orchestra.run.smithi050.stdout:(114/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 798 kB/s | 839 kB 00:01 2024-04-26T21:22:04.906 INFO:teuthology.orchestra.run.smithi142.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-04-26T21:22:04.910 INFO:teuthology.orchestra.run.smithi046.stdout:(108/152): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-04-26T21:22:04.923 INFO:teuthology.orchestra.run.smithi050.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 321 kB/s | 32 kB 00:00 2024-04-26T21:22:04.935 INFO:teuthology.orchestra.run.smithi046.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 388 kB/s | 49 kB 00:00 2024-04-26T21:22:04.957 INFO:teuthology.orchestra.run.smithi050.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 150 kB/s | 15 kB 00:00 2024-04-26T21:22:05.010 INFO:teuthology.orchestra.run.smithi046.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 674 kB/s | 67 kB 00:00 2024-04-26T21:22:05.061 INFO:teuthology.orchestra.run.smithi046.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-04-26T21:22:05.128 INFO:teuthology.orchestra.run.smithi046.stdout:(112/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-04-26T21:22:05.191 INFO:teuthology.orchestra.run.smithi050.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 644 kB/s | 172 kB 00:00 2024-04-26T21:22:05.228 INFO:teuthology.orchestra.run.smithi046.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 435 kB/s | 43 kB 00:00 2024-04-26T21:22:05.261 INFO:teuthology.orchestra.run.smithi046.stdout:(114/152): python3-asyncssh-2.13.2-5.el9.noarch 2.7 MB/s | 548 kB 00:00 2024-04-26T21:22:05.306 INFO:teuthology.orchestra.run.smithi046.stdout:(115/152): protobuf-compiler-3.14.0-13.el9.x86_ 476 kB/s | 863 kB 00:01 2024-04-26T21:22:05.331 INFO:teuthology.orchestra.run.smithi046.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 313 kB/s | 32 kB 00:00 2024-04-26T21:22:05.356 INFO:teuthology.orchestra.run.smithi046.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 159 kB/s | 15 kB 00:00 2024-04-26T21:22:05.466 INFO:teuthology.orchestra.run.smithi050.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 705 kB/s | 358 kB 00:00 2024-04-26T21:22:05.481 INFO:teuthology.orchestra.run.smithi046.stdout:(118/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-04-26T21:22:05.523 INFO:teuthology.orchestra.run.smithi046.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 1.8 MB/s | 358 kB 00:00 2024-04-26T21:22:05.574 INFO:teuthology.orchestra.run.smithi050.stdout:(119/152): python3-colorama-0.4.4-12.el9.noarch 326 kB/s | 35 kB 00:00 2024-04-26T21:22:05.598 INFO:teuthology.orchestra.run.smithi046.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 302 kB/s | 35 kB 00:00 2024-04-26T21:22:05.608 INFO:teuthology.orchestra.run.smithi050.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 727 kB/s | 303 kB 00:00 2024-04-26T21:22:05.624 INFO:teuthology.orchestra.run.smithi046.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 2.0 MB/s | 206 kB 00:00 2024-04-26T21:22:05.741 INFO:teuthology.orchestra.run.smithi046.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-04-26T21:22:05.783 INFO:teuthology.orchestra.run.smithi046.stdout:(123/152): python3-cheroot-8.6.0-4.el9.noarch.r 360 kB/s | 172 kB 00:00 2024-04-26T21:22:05.841 INFO:teuthology.orchestra.run.smithi046.stdout:(124/152): python3-grpcio-1.46.7-10.el9.x86_64. 8.4 MB/s | 2.0 MB 00:00 2024-04-26T21:22:05.891 INFO:teuthology.orchestra.run.smithi046.stdout:(125/152): python3-jaraco-8.2.1-3.el9.noarch.rp 71 kB/s | 11 kB 00:00 2024-04-26T21:22:05.917 INFO:teuthology.orchestra.run.smithi046.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 132 kB/s | 18 kB 00:00 2024-04-26T21:22:05.942 INFO:teuthology.orchestra.run.smithi046.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 231 kB/s | 23 kB 00:00 2024-04-26T21:22:05.942 INFO:teuthology.orchestra.run.smithi050.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 559 kB/s | 206 kB 00:00 2024-04-26T21:22:05.984 INFO:teuthology.orchestra.run.smithi046.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-04-26T21:22:06.017 INFO:teuthology.orchestra.run.smithi046.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-04-26T21:22:06.059 INFO:teuthology.orchestra.run.smithi046.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 8.7 MB/s | 1.0 MB 00:00 2024-04-26T21:22:06.085 INFO:teuthology.orchestra.run.smithi046.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 460 kB/s | 46 kB 00:00 2024-04-26T21:22:06.135 INFO:teuthology.orchestra.run.smithi046.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 673 kB/s | 79 kB 00:00 2024-04-26T21:22:06.152 INFO:teuthology.orchestra.run.smithi046.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 627 kB/s | 58 kB 00:00 2024-04-26T21:22:06.194 INFO:teuthology.orchestra.run.smithi046.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-04-26T21:22:06.225 INFO:teuthology.orchestra.run.smithi050.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 512 kB/s | 144 kB 00:00 2024-04-26T21:22:06.227 INFO:teuthology.orchestra.run.smithi046.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 178 kB/s | 16 kB 00:00 2024-04-26T21:22:06.252 INFO:teuthology.orchestra.run.smithi046.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 897 kB/s | 90 kB 00:00 2024-04-26T21:22:06.286 INFO:teuthology.orchestra.run.smithi046.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 307 kB/s | 28 kB 00:00 2024-04-26T21:22:06.325 INFO:teuthology.orchestra.run.smithi050.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 107 kB/s | 11 kB 00:00 2024-04-26T21:22:06.328 INFO:teuthology.orchestra.run.smithi046.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 306 kB/s | 31 kB 00:00 2024-04-26T21:22:06.353 INFO:teuthology.orchestra.run.smithi046.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-04-26T21:22:06.386 INFO:teuthology.orchestra.run.smithi046.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 588 kB/s | 59 kB 00:00 2024-04-26T21:22:06.426 INFO:teuthology.orchestra.run.smithi050.stdout:(124/152): python3-jaraco-classes-3.2.1-5.el9.n 176 kB/s | 18 kB 00:00 2024-04-26T21:22:06.437 INFO:teuthology.orchestra.run.smithi046.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 329 kB/s | 36 kB 00:00 2024-04-26T21:22:06.462 INFO:teuthology.orchestra.run.smithi046.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 468 kB/s | 51 kB 00:00 2024-04-26T21:22:06.487 INFO:teuthology.orchestra.run.smithi046.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-04-26T21:22:06.518 INFO:teuthology.orchestra.run.smithi050.stdout:(125/152): python3-jaraco-collections-3.0.0-8.e 252 kB/s | 23 kB 00:00 2024-04-26T21:22:06.563 INFO:teuthology.orchestra.run.smithi046.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 715 kB/s | 90 kB 00:00 2024-04-26T21:22:06.588 INFO:teuthology.orchestra.run.smithi046.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 199 kB/s | 20 kB 00:00 2024-04-26T21:22:06.610 INFO:teuthology.orchestra.run.smithi050.stdout:(126/152): python3-jaraco-functools-3.5.0-2.el9 212 kB/s | 19 kB 00:00 2024-04-26T21:22:06.613 INFO:teuthology.orchestra.run.smithi046.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 2.8 MB/s | 427 kB 00:00 2024-04-26T21:22:06.702 INFO:teuthology.orchestra.run.smithi050.stdout:(127/152): python3-jaraco-text-3.2.0-6.el9.noar 214 kB/s | 20 kB 00:00 2024-04-26T21:22:06.772 INFO:teuthology.orchestra.run.smithi046.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 917 kB/s | 192 kB 00:00 2024-04-26T21:22:06.806 INFO:teuthology.orchestra.run.smithi046.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-04-26T21:22:06.981 INFO:teuthology.orchestra.run.smithi046.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 7.7 MB/s | 1.3 MB 00:00 2024-04-26T21:22:07.007 INFO:teuthology.orchestra.run.smithi142.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T21:22:07.007 INFO:teuthology.orchestra.run.smithi142.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T21:22:07.018 INFO:teuthology.orchestra.run.smithi142.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T21:22:07.019 INFO:teuthology.orchestra.run.smithi142.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T21:22:07.020 INFO:teuthology.orchestra.run.smithi142.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T21:22:07.020 INFO:teuthology.orchestra.run.smithi142.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T21:22:07.022 INFO:teuthology.orchestra.run.smithi142.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T21:22:07.024 INFO:teuthology.orchestra.run.smithi142.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T21:22:07.025 INFO:teuthology.orchestra.run.smithi142.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T21:22:07.081 INFO:teuthology.orchestra.run.smithi046.stdout:(150/152): librados2-19.0.0-3306.gbc612ffb.el9. 7.3 MB/s | 3.4 MB 00:00 2024-04-26T21:22:07.085 INFO:teuthology.orchestra.run.smithi142.stdout:Dependencies resolved. 2024-04-26T21:22:07.100 INFO:teuthology.orchestra.run.smithi142.stdout:======================================================================================= 2024-04-26T21:22:07.100 INFO:teuthology.orchestra.run.smithi142.stdout: Package Arch Version Repository Size 2024-04-26T21:22:07.100 INFO:teuthology.orchestra.run.smithi142.stdout:======================================================================================= 2024-04-26T21:22:07.101 INFO:teuthology.orchestra.run.smithi142.stdout:Installing: 2024-04-26T21:22:07.101 INFO:teuthology.orchestra.run.smithi142.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T21:22:07.101 INFO:teuthology.orchestra.run.smithi142.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-26T21:22:07.101 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T21:22:07.101 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-26T21:22:07.101 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-26T21:22:07.101 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-26T21:22:07.101 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-26T21:22:07.101 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-26T21:22:07.101 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-26T21:22:07.101 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-26T21:22:07.101 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-26T21:22:07.101 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-26T21:22:07.101 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-26T21:22:07.102 INFO:teuthology.orchestra.run.smithi142.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-26T21:22:07.102 INFO:teuthology.orchestra.run.smithi142.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-26T21:22:07.102 INFO:teuthology.orchestra.run.smithi142.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-26T21:22:07.102 INFO:teuthology.orchestra.run.smithi142.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-26T21:22:07.102 INFO:teuthology.orchestra.run.smithi142.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T21:22:07.102 INFO:teuthology.orchestra.run.smithi142.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T21:22:07.102 INFO:teuthology.orchestra.run.smithi142.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T21:22:07.102 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-26T21:22:07.102 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-26T21:22:07.102 INFO:teuthology.orchestra.run.smithi142.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-26T21:22:07.102 INFO:teuthology.orchestra.run.smithi142.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T21:22:07.102 INFO:teuthology.orchestra.run.smithi142.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T21:22:07.102 INFO:teuthology.orchestra.run.smithi142.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T21:22:07.103 INFO:teuthology.orchestra.run.smithi142.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T21:22:07.103 INFO:teuthology.orchestra.run.smithi142.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T21:22:07.103 INFO:teuthology.orchestra.run.smithi142.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T21:22:07.103 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-26T21:22:07.103 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-26T21:22:07.103 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-26T21:22:07.103 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-26T21:22:07.103 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-26T21:22:07.103 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T21:22:07.103 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-26T21:22:07.103 INFO:teuthology.orchestra.run.smithi142.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T21:22:07.103 INFO:teuthology.orchestra.run.smithi142.stdout:Upgrading: 2024-04-26T21:22:07.103 INFO:teuthology.orchestra.run.smithi142.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-26T21:22:07.103 INFO:teuthology.orchestra.run.smithi142.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T21:22:07.104 INFO:teuthology.orchestra.run.smithi142.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T21:22:07.104 INFO:teuthology.orchestra.run.smithi142.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T21:22:07.104 INFO:teuthology.orchestra.run.smithi142.stdout:Installing dependencies: 2024-04-26T21:22:07.104 INFO:teuthology.orchestra.run.smithi142.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T21:22:07.104 INFO:teuthology.orchestra.run.smithi142.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T21:22:07.104 INFO:teuthology.orchestra.run.smithi142.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T21:22:07.104 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-26T21:22:07.104 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-26T21:22:07.104 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-26T21:22:07.104 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-26T21:22:07.104 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-26T21:22:07.104 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-26T21:22:07.104 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-26T21:22:07.104 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-26T21:22:07.105 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T21:22:07.105 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T21:22:07.105 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T21:22:07.105 INFO:teuthology.orchestra.run.smithi142.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T21:22:07.105 INFO:teuthology.orchestra.run.smithi142.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T21:22:07.105 INFO:teuthology.orchestra.run.smithi142.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T21:22:07.105 INFO:teuthology.orchestra.run.smithi142.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T21:22:07.105 INFO:teuthology.orchestra.run.smithi142.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T21:22:07.105 INFO:teuthology.orchestra.run.smithi142.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-26T21:22:07.105 INFO:teuthology.orchestra.run.smithi142.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T21:22:07.105 INFO:teuthology.orchestra.run.smithi142.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T21:22:07.105 INFO:teuthology.orchestra.run.smithi142.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T21:22:07.105 INFO:teuthology.orchestra.run.smithi142.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T21:22:07.105 INFO:teuthology.orchestra.run.smithi142.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T21:22:07.106 INFO:teuthology.orchestra.run.smithi142.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-26T21:22:07.106 INFO:teuthology.orchestra.run.smithi142.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T21:22:07.106 INFO:teuthology.orchestra.run.smithi142.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T21:22:07.106 INFO:teuthology.orchestra.run.smithi142.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T21:22:07.106 INFO:teuthology.orchestra.run.smithi142.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T21:22:07.106 INFO:teuthology.orchestra.run.smithi142.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T21:22:07.106 INFO:teuthology.orchestra.run.smithi142.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T21:22:07.106 INFO:teuthology.orchestra.run.smithi142.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T21:22:07.106 INFO:teuthology.orchestra.run.smithi142.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T21:22:07.107 INFO:teuthology.orchestra.run.smithi142.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T21:22:07.107 INFO:teuthology.orchestra.run.smithi142.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T21:22:07.107 INFO:teuthology.orchestra.run.smithi142.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T21:22:07.107 INFO:teuthology.orchestra.run.smithi142.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T21:22:07.107 INFO:teuthology.orchestra.run.smithi142.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T21:22:07.107 INFO:teuthology.orchestra.run.smithi142.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T21:22:07.107 INFO:teuthology.orchestra.run.smithi142.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T21:22:07.107 INFO:teuthology.orchestra.run.smithi142.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T21:22:07.107 INFO:teuthology.orchestra.run.smithi142.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T21:22:07.107 INFO:teuthology.orchestra.run.smithi142.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T21:22:07.107 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T21:22:07.107 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-26T21:22:07.107 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-26T21:22:07.107 INFO:teuthology.orchestra.run.smithi142.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T21:22:07.108 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T21:22:07.108 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T21:22:07.108 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T21:22:07.108 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T21:22:07.108 INFO:teuthology.orchestra.run.smithi142.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T21:22:07.108 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T21:22:07.108 INFO:teuthology.orchestra.run.smithi142.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T21:22:07.108 INFO:teuthology.orchestra.run.smithi142.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T21:22:07.108 INFO:teuthology.orchestra.run.smithi142.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T21:22:07.108 INFO:teuthology.orchestra.run.smithi142.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T21:22:07.108 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T21:22:07.108 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T21:22:07.108 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T21:22:07.108 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T21:22:07.109 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T21:22:07.109 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T21:22:07.109 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T21:22:07.109 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T21:22:07.109 INFO:teuthology.orchestra.run.smithi142.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T21:22:07.109 INFO:teuthology.orchestra.run.smithi142.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T21:22:07.109 INFO:teuthology.orchestra.run.smithi142.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T21:22:07.109 INFO:teuthology.orchestra.run.smithi142.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T21:22:07.109 INFO:teuthology.orchestra.run.smithi142.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T21:22:07.109 INFO:teuthology.orchestra.run.smithi142.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T21:22:07.109 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T21:22:07.109 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T21:22:07.109 INFO:teuthology.orchestra.run.smithi142.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T21:22:07.109 INFO:teuthology.orchestra.run.smithi142.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T21:22:07.110 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T21:22:07.110 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T21:22:07.110 INFO:teuthology.orchestra.run.smithi142.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T21:22:07.110 INFO:teuthology.orchestra.run.smithi142.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T21:22:07.110 INFO:teuthology.orchestra.run.smithi142.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T21:22:07.110 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T21:22:07.110 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T21:22:07.110 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T21:22:07.110 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T21:22:07.110 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T21:22:07.110 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T21:22:07.110 INFO:teuthology.orchestra.run.smithi142.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T21:22:07.110 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T21:22:07.111 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T21:22:07.111 INFO:teuthology.orchestra.run.smithi142.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T21:22:07.111 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T21:22:07.111 INFO:teuthology.orchestra.run.smithi142.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T21:22:07.111 INFO:teuthology.orchestra.run.smithi142.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T21:22:07.111 INFO:teuthology.orchestra.run.smithi142.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T21:22:07.111 INFO:teuthology.orchestra.run.smithi142.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T21:22:07.111 INFO:teuthology.orchestra.run.smithi142.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T21:22:07.111 INFO:teuthology.orchestra.run.smithi142.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T21:22:07.111 INFO:teuthology.orchestra.run.smithi142.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T21:22:07.111 INFO:teuthology.orchestra.run.smithi142.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T21:22:07.111 INFO:teuthology.orchestra.run.smithi142.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T21:22:07.111 INFO:teuthology.orchestra.run.smithi142.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T21:22:07.111 INFO:teuthology.orchestra.run.smithi142.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T21:22:07.112 INFO:teuthology.orchestra.run.smithi142.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T21:22:07.112 INFO:teuthology.orchestra.run.smithi142.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T21:22:07.112 INFO:teuthology.orchestra.run.smithi142.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T21:22:07.112 INFO:teuthology.orchestra.run.smithi142.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T21:22:07.112 INFO:teuthology.orchestra.run.smithi142.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T21:22:07.112 INFO:teuthology.orchestra.run.smithi142.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T21:22:07.112 INFO:teuthology.orchestra.run.smithi142.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T21:22:07.112 INFO:teuthology.orchestra.run.smithi142.stdout:Installing weak dependencies: 2024-04-26T21:22:07.112 INFO:teuthology.orchestra.run.smithi142.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T21:22:07.112 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-04-26T21:22:07.112 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction Summary 2024-04-26T21:22:07.112 INFO:teuthology.orchestra.run.smithi142.stdout:======================================================================================= 2024-04-26T21:22:07.112 INFO:teuthology.orchestra.run.smithi142.stdout:Install 148 Packages 2024-04-26T21:22:07.112 INFO:teuthology.orchestra.run.smithi142.stdout:Upgrade 4 Packages 2024-04-26T21:22:07.113 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-04-26T21:22:07.113 INFO:teuthology.orchestra.run.smithi142.stdout:Total download size: 222 M 2024-04-26T21:22:07.113 INFO:teuthology.orchestra.run.smithi142.stdout:Downloading Packages: 2024-04-26T21:22:07.131 INFO:teuthology.orchestra.run.smithi046.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 16 MB/s | 2.4 MB 00:00 2024-04-26T21:22:07.257 INFO:teuthology.orchestra.run.smithi046.stdout:(152/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 6.5 MB/s | 3.1 MB 00:00 2024-04-26T21:22:07.259 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-04-26T21:22:07.260 INFO:teuthology.orchestra.run.smithi046.stdout:Total 14 MB/s | 222 MB 00:15 2024-04-26T21:22:07.965 INFO:teuthology.orchestra.run.smithi142.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 31 kB/s | 6.6 kB 00:00 2024-04-26T21:22:08.010 INFO:teuthology.orchestra.run.smithi050.stdout:(128/152): python3-kubernetes-26.1.0-2.el9.noar 799 kB/s | 1.0 MB 00:01 2024-04-26T21:22:08.052 INFO:teuthology.orchestra.run.smithi050.stdout:(129/152): python3-grpcio-1.46.7-10.el9.x86_64. 854 kB/s | 2.0 MB 00:02 2024-04-26T21:22:08.153 INFO:teuthology.orchestra.run.smithi050.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 326 kB/s | 46 kB 00:00 2024-04-26T21:22:08.253 INFO:teuthology.orchestra.run.smithi050.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 393 kB/s | 79 kB 00:00 2024-04-26T21:22:08.354 INFO:teuthology.orchestra.run.smithi050.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 288 kB/s | 58 kB 00:00 2024-04-26T21:22:08.402 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-04-26T21:22:08.445 INFO:teuthology.orchestra.run.smithi050.stdout:(133/152): python3-portend-3.1.0-2.el9.noarch.r 179 kB/s | 16 kB 00:00 2024-04-26T21:22:08.507 INFO:teuthology.orchestra.run.smithi142.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.6 MB/s | 889 kB 00:00 2024-04-26T21:22:08.546 INFO:teuthology.orchestra.run.smithi050.stdout:(134/152): libarrow-9.0.0-11.el9.x86_64.rpm 852 kB/s | 4.4 MB 00:05 2024-04-26T21:22:08.561 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-04-26T21:22:08.561 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-04-26T21:22:08.624 INFO:teuthology.orchestra.run.smithi142.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 1.2 MB/s | 145 kB 00:00 2024-04-26T21:22:08.637 INFO:teuthology.orchestra.run.smithi050.stdout:(135/152): python3-pyperclip-1.8.2-2.el9.noarch 308 kB/s | 28 kB 00:00 2024-04-26T21:22:08.663 INFO:teuthology.orchestra.run.smithi050.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 414 kB/s | 90 kB 00:00 2024-04-26T21:22:08.713 INFO:teuthology.orchestra.run.smithi050.stdout:(137/152): python3-pecan-1.4.2-3.el9.noarch.rpm 593 kB/s | 272 kB 00:00 2024-04-26T21:22:08.730 INFO:teuthology.orchestra.run.smithi050.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 335 kB/s | 31 kB 00:00 2024-04-26T21:22:08.822 INFO:teuthology.orchestra.run.smithi050.stdout:(139/152): python3-tempora-5.0.0-2.el9.noarch.r 390 kB/s | 36 kB 00:00 2024-04-26T21:22:08.889 INFO:teuthology.orchestra.run.smithi050.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 337 kB/s | 59 kB 00:00 2024-04-26T21:22:08.914 INFO:teuthology.orchestra.run.smithi050.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 553 kB/s | 51 kB 00:00 2024-04-26T21:22:08.931 INFO:teuthology.orchestra.run.smithi050.stdout:(142/152): python3-routes-2.5.1-5.el9.noarch.rp 703 kB/s | 188 kB 00:00 2024-04-26T21:22:09.050 INFO:teuthology.orchestra.run.smithi142.stdout:(4/152): ceph-common-19.0.0-3306.gbc612ffb.el9. 16 MB/s | 21 MB 00:01 2024-04-26T21:22:09.090 INFO:teuthology.orchestra.run.smithi050.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 512 kB/s | 90 kB 00:00 2024-04-26T21:22:09.166 INFO:teuthology.orchestra.run.smithi142.stdout:(5/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 13 MB/s | 1.5 MB 00:00 2024-04-26T21:22:09.190 INFO:teuthology.orchestra.run.smithi050.stdout:(144/152): python3-zc-lockfile-2.0-10.el9.noarc 200 kB/s | 20 kB 00:00 2024-04-26T21:22:09.232 INFO:teuthology.orchestra.run.smithi050.stdout:(145/152): python3-webob-1.8.7-6.el9.noarch.rpm 670 kB/s | 230 kB 00:00 2024-04-26T21:22:09.408 INFO:teuthology.orchestra.run.smithi142.stdout:(6/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 3.0 MB/s | 2.4 MB 00:00 2024-04-26T21:22:09.449 INFO:teuthology.orchestra.run.smithi050.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 741 kB/s | 192 kB 00:00 2024-04-26T21:22:09.475 INFO:teuthology.orchestra.run.smithi142.stdout:(7/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 15 MB/s | 4.7 MB 00:00 2024-04-26T21:22:09.674 INFO:teuthology.orchestra.run.smithi050.stdout:(147/152): python3-werkzeug-2.0.3-3.el9.1.noarc 575 kB/s | 427 kB 00:00 2024-04-26T21:22:10.050 INFO:teuthology.orchestra.run.smithi142.stdout:(8/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 2.3 MB/s | 5.4 MB 00:02 2024-04-26T21:22:10.176 INFO:teuthology.orchestra.run.smithi142.stdout:(9/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el9 15 MB/s | 11 MB 00:00 2024-04-26T21:22:10.200 INFO:teuthology.orchestra.run.smithi142.stdout:(10/152): ceph-selinux-19.0.0-3306.gbc612ffb.el 168 kB/s | 25 kB 00:00 2024-04-26T21:22:10.224 INFO:teuthology.orchestra.run.smithi050.stdout:(148/152): librados2-19.0.0-3306.gbc612ffb.el9. 4.4 MB/s | 3.4 MB 00:00 2024-04-26T21:22:10.368 INFO:teuthology.orchestra.run.smithi142.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 1.3 MB/s | 225 kB 00:00 2024-04-26T21:22:10.374 INFO:teuthology.orchestra.run.smithi050.stdout:(149/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 4.5 MB/s | 3.1 MB 00:00 2024-04-26T21:22:10.433 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-04-26T21:22:10.433 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-04-26T21:22:10.434 INFO:teuthology.orchestra.run.smithi050.stdout:(150/152): openssl-3.2.1-1.el9.x86_64.rpm 6.5 MB/s | 1.3 MB 00:00 2024-04-26T21:22:10.442 INFO:teuthology.orchestra.run.smithi142.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 361 kB/s | 27 kB 00:00 2024-04-26T21:22:10.518 INFO:teuthology.orchestra.run.smithi142.stdout:(13/152): libcephfs-devel-19.0.0-3306.gbc612ffb 423 kB/s | 32 kB 00:00 2024-04-26T21:22:10.575 INFO:teuthology.orchestra.run.smithi050.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 12 MB/s | 2.4 MB 00:00 2024-04-26T21:22:10.744 INFO:teuthology.orchestra.run.smithi142.stdout:(14/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 3.2 MB/s | 747 kB 00:00 2024-04-26T21:22:10.836 INFO:teuthology.orchestra.run.smithi142.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 1.7 MB/s | 163 kB 00:00 2024-04-26T21:22:10.919 INFO:teuthology.orchestra.run.smithi142.stdout:(16/152): librados-devel-19.0.0-3306.gbc612ffb. 1.5 MB/s | 127 kB 00:00 2024-04-26T21:22:11.111 INFO:teuthology.orchestra.run.smithi142.stdout:(17/152): libradosstriper1-19.0.0-3306.gbc612ff 2.6 MB/s | 501 kB 00:00 2024-04-26T21:22:11.283 INFO:teuthology.orchestra.run.smithi050.stdout:(152/152): thrift-0.15.0-2.el9.x86_64.rpm 789 kB/s | 1.6 MB 00:02 2024-04-26T21:22:11.291 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-04-26T21:22:11.291 INFO:teuthology.orchestra.run.smithi050.stdout:Total 10 MB/s | 222 MB 00:21 2024-04-26T21:22:12.433 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-04-26T21:22:12.561 INFO:teuthology.orchestra.run.smithi142.stdout:(18/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 3.7 MB/s | 5.4 MB 00:01 2024-04-26T21:22:12.592 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-04-26T21:22:12.592 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-04-26T21:22:12.970 INFO:teuthology.orchestra.run.smithi142.stdout:(19/152): ceph-test-19.0.0-3306.gbc612ffb.el9.x 18 MB/s | 49 MB 00:02 2024-04-26T21:22:13.011 INFO:teuthology.orchestra.run.smithi142.stdout:(20/152): python3-ceph-argparse-19.0.0-3306.gbc 100 kB/s | 45 kB 00:00 2024-04-26T21:22:13.045 INFO:teuthology.orchestra.run.smithi142.stdout:(21/152): python3-ceph-common-19.0.0-3306.gbc61 1.8 MB/s | 135 kB 00:00 2024-04-26T21:22:13.095 INFO:teuthology.orchestra.run.smithi142.stdout:(22/152): python3-rados-19.0.0-3306.gbc612ffb.e 6.3 MB/s | 321 kB 00:00 2024-04-26T21:22:13.120 INFO:teuthology.orchestra.run.smithi142.stdout:(23/152): python3-cephfs-19.0.0-3306.gbc612ffb. 1.5 MB/s | 161 kB 00:00 2024-04-26T21:22:13.145 INFO:teuthology.orchestra.run.smithi142.stdout:(24/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 5.8 MB/s | 298 kB 00:00 2024-04-26T21:22:13.179 INFO:teuthology.orchestra.run.smithi142.stdout:(25/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 1.7 MB/s | 99 kB 00:00 2024-04-26T21:22:13.204 INFO:teuthology.orchestra.run.smithi142.stdout:(26/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.4 MB/s | 85 kB 00:00 2024-04-26T21:22:13.254 INFO:teuthology.orchestra.run.smithi142.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 3.3 MB/s | 171 kB 00:00 2024-04-26T21:22:13.305 INFO:teuthology.orchestra.run.smithi142.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3306.g 684 kB/s | 34 kB 00:00 2024-04-26T21:22:13.364 INFO:teuthology.orchestra.run.smithi142.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 2.4 MB/s | 142 kB 00:00 2024-04-26T21:22:13.606 INFO:teuthology.orchestra.run.smithi142.stdout:(30/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x8 4.0 MB/s | 17 MB 00:04 2024-04-26T21:22:13.665 INFO:teuthology.orchestra.run.smithi142.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 12 MB/s | 3.5 MB 00:00 2024-04-26T21:22:13.732 INFO:teuthology.orchestra.run.smithi142.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 3.6 MB/s | 249 kB 00:00 2024-04-26T21:22:13.773 INFO:teuthology.orchestra.run.smithi142.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 1.2 MB/s | 49 kB 00:00 2024-04-26T21:22:13.815 INFO:teuthology.orchestra.run.smithi142.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3306.gb 396 kB/s | 16 kB 00:00 2024-04-26T21:22:13.874 INFO:teuthology.orchestra.run.smithi142.stdout:(35/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 4.6 MB/s | 273 kB 00:00 2024-04-26T21:22:13.924 INFO:teuthology.orchestra.run.smithi142.stdout:(36/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 4.1 MB/s | 3.1 MB 00:00 2024-04-26T21:22:13.958 INFO:teuthology.orchestra.run.smithi142.stdout:(37/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 8.9 MB/s | 762 kB 00:00 2024-04-26T21:22:13.983 INFO:teuthology.orchestra.run.smithi142.stdout:(38/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 313 kB/s | 18 kB 00:00 2024-04-26T21:22:14.383 INFO:teuthology.orchestra.run.smithi142.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 189 kB/s | 80 kB 00:00 2024-04-26T21:22:14.472 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-04-26T21:22:14.472 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-04-26T21:22:14.809 INFO:teuthology.orchestra.run.smithi142.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 460 kB/s | 195 kB 00:00 2024-04-26T21:22:14.893 INFO:teuthology.orchestra.run.smithi142.stdout:(41/152): ceph-mgr-diskprediction-local-19.0.0- 5.8 MB/s | 7.4 MB 00:01 2024-04-26T21:22:15.559 INFO:teuthology.orchestra.run.smithi142.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 338 kB/s | 253 kB 00:00 2024-04-26T21:22:15.609 INFO:teuthology.orchestra.run.smithi142.stdout:(43/152): libgfortran-11.4.1-3.el9.x86_64.rpm 494 kB/s | 803 kB 00:01 2024-04-26T21:22:15.993 INFO:teuthology.orchestra.run.smithi142.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 246 kB/s | 106 kB 00:00 2024-04-26T21:22:16.185 INFO:teuthology.orchestra.run.smithi142.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 235 kB/s | 135 kB 00:00 2024-04-26T21:22:16.426 INFO:teuthology.orchestra.run.smithi142.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 288 kB/s | 125 kB 00:00 2024-04-26T21:22:16.619 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-04-26T21:22:16.711 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T21:22:16.735 INFO:teuthology.orchestra.run.smithi142.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 2.2 MB/s | 689 kB 00:00 2024-04-26T21:22:16.758 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T21:22:16.812 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T21:22:16.862 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T21:22:16.894 INFO:teuthology.orchestra.run.smithi142.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 5.8 MB/s | 943 kB 00:00 2024-04-26T21:22:16.927 INFO:teuthology.orchestra.run.smithi142.stdout:(49/152): python3-urllib3-1.26.5-5.el9.noarch.r 289 kB/s | 215 kB 00:00 2024-04-26T21:22:16.952 INFO:teuthology.orchestra.run.smithi142.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-04-26T21:22:17.003 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T21:22:17.011 INFO:teuthology.orchestra.run.smithi142.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 5.3 MB/s | 313 kB 00:00 2024-04-26T21:22:17.045 INFO:teuthology.orchestra.run.smithi142.stdout:(52/152): python3-cryptography-36.0.1-4.el9.x86 583 kB/s | 1.2 MB 00:02 2024-04-26T21:22:17.048 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T21:22:17.070 INFO:teuthology.orchestra.run.smithi142.stdout:(53/152): elfutils-libelf-devel-0.190-2.el9.x86 151 kB/s | 21 kB 00:00 2024-04-26T21:22:17.095 INFO:teuthology.orchestra.run.smithi142.stdout:(54/152): flexiblas-3.0.4-8.el9.x86_64.rpm 390 kB/s | 33 kB 00:00 2024-04-26T21:22:17.154 INFO:teuthology.orchestra.run.smithi142.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 192 kB/s | 11 kB 00:00 2024-04-26T21:22:17.187 INFO:teuthology.orchestra.run.smithi142.stdout:(56/152): flexiblas-openblas-openmp-3.0.4-8.el9 155 kB/s | 18 kB 00:00 2024-04-26T21:22:17.213 INFO:teuthology.orchestra.run.smithi142.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-04-26T21:22:17.305 INFO:teuthology.orchestra.run.smithi142.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 387 kB/s | 45 kB 00:00 2024-04-26T21:22:17.329 INFO:teuthology.orchestra.run.smithi142.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-04-26T21:22:17.388 INFO:teuthology.orchestra.run.smithi142.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 437 kB/s | 25 kB 00:00 2024-04-26T21:22:17.391 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T21:22:17.413 INFO:teuthology.orchestra.run.smithi142.stdout:(61/152): libtool-2.4.6-45.el9.x86_64.rpm 5.2 MB/s | 582 kB 00:00 2024-04-26T21:22:17.447 INFO:teuthology.orchestra.run.smithi142.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 811 kB/s | 47 kB 00:00 2024-04-26T21:22:17.473 INFO:teuthology.orchestra.run.smithi142.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-04-26T21:22:17.495 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T21:22:17.531 INFO:teuthology.orchestra.run.smithi142.stdout:(64/152): openblas-0.3.21-2.el9.x86_64.rpm 603 kB/s | 35 kB 00:00 2024-04-26T21:22:17.556 INFO:teuthology.orchestra.run.smithi142.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-04-26T21:22:17.598 INFO:teuthology.orchestra.run.smithi142.stdout:(66/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.4 MB/s | 3.0 MB 00:00 2024-04-26T21:22:17.629 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T21:22:17.655 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T21:22:17.748 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T21:22:17.782 INFO:teuthology.orchestra.run.smithi142.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 8.7 MB/s | 1.6 MB 00:00 2024-04-26T21:22:17.784 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T21:22:17.891 INFO:teuthology.orchestra.run.smithi142.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 8.2 MB/s | 911 kB 00:00 2024-04-26T21:22:17.944 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T21:22:17.949 INFO:teuthology.orchestra.run.smithi142.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-04-26T21:22:18.008 INFO:teuthology.orchestra.run.smithi142.stdout:(70/152): openblas-openmp-0.3.21-2.el9.x86_64.r 10 MB/s | 4.7 MB 00:00 2024-04-26T21:22:18.040 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T21:22:18.068 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T21:22:18.100 INFO:teuthology.orchestra.run.smithi142.stdout:(71/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 8.1 MB/s | 4.4 MB 00:00 2024-04-26T21:22:18.133 INFO:teuthology.orchestra.run.smithi142.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 8.1 MB/s | 1.0 MB 00:00 2024-04-26T21:22:18.158 INFO:teuthology.orchestra.run.smithi142.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-26T21:22:18.173 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T21:22:18.201 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T21:22:18.217 INFO:teuthology.orchestra.run.smithi142.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 4.1 MB/s | 244 kB 00:00 2024-04-26T21:22:18.265 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T21:22:18.286 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T21:22:18.292 INFO:teuthology.orchestra.run.smithi142.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 3.2 MB/s | 248 kB 00:00 2024-04-26T21:22:18.312 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T21:22:18.359 INFO:teuthology.orchestra.run.smithi142.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 385 kB/s | 26 kB 00:00 2024-04-26T21:22:18.460 INFO:teuthology.orchestra.run.smithi142.stdout:(77/152): postgresql-server-13.14-1.el9.x86_64. 11 MB/s | 5.8 MB 00:00 2024-04-26T21:22:18.462 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T21:22:18.486 INFO:teuthology.orchestra.run.smithi142.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 153 kB/s | 19 kB 00:00 2024-04-26T21:22:18.494 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T21:22:18.560 INFO:teuthology.orchestra.run.smithi142.stdout:(79/152): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-04-26T21:22:18.585 INFO:teuthology.orchestra.run.smithi142.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 346 kB/s | 35 kB 00:00 2024-04-26T21:22:18.605 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T21:22:18.610 INFO:teuthology.orchestra.run.smithi142.stdout:(81/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-04-26T21:22:18.644 INFO:teuthology.orchestra.run.smithi142.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 250 kB 00:00 2024-04-26T21:22:18.677 INFO:teuthology.orchestra.run.smithi142.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 3.2 MB/s | 222 kB 00:00 2024-04-26T21:22:18.694 INFO:teuthology.orchestra.run.smithi142.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-26T21:22:18.701 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T21:22:18.736 INFO:teuthology.orchestra.run.smithi142.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 723 kB/s | 42 kB 00:00 2024-04-26T21:22:18.757 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T21:22:18.761 INFO:teuthology.orchestra.run.smithi142.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 3.9 MB/s | 269 kB 00:00 2024-04-26T21:22:18.795 INFO:teuthology.orchestra.run.smithi142.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-04-26T21:22:18.796 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T21:22:18.820 INFO:teuthology.orchestra.run.smithi142.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 4.7 MB/s | 279 kB 00:00 2024-04-26T21:22:18.879 INFO:teuthology.orchestra.run.smithi142.stdout:(89/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 5.1 MB 00:00 2024-04-26T21:22:18.879 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T21:22:18.904 INFO:teuthology.orchestra.run.smithi142.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 469 kB/s | 51 kB 00:00 2024-04-26T21:22:18.920 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T21:22:18.929 INFO:teuthology.orchestra.run.smithi142.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 494 kB/s | 54 kB 00:00 2024-04-26T21:22:18.972 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T21:22:18.988 INFO:teuthology.orchestra.run.smithi142.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 715 kB/s | 42 kB 00:00 2024-04-26T21:22:19.012 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T21:22:19.013 INFO:teuthology.orchestra.run.smithi142.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 529 kB/s | 44 kB 00:00 2024-04-26T21:22:19.040 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T21:22:19.046 INFO:teuthology.orchestra.run.smithi142.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-04-26T21:22:19.072 INFO:teuthology.orchestra.run.smithi142.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 613 kB/s | 36 kB 00:00 2024-04-26T21:22:19.106 INFO:teuthology.orchestra.run.smithi142.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 807 kB/s | 47 kB 00:00 2024-04-26T21:22:19.112 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T21:22:19.139 INFO:teuthology.orchestra.run.smithi142.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 305 kB 00:00 2024-04-26T21:22:19.165 INFO:teuthology.orchestra.run.smithi142.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 984 kB/s | 58 kB 00:00 2024-04-26T21:22:19.172 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T21:22:19.198 INFO:teuthology.orchestra.run.smithi142.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-04-26T21:22:19.201 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T21:22:19.224 INFO:teuthology.orchestra.run.smithi142.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-26T21:22:19.245 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T21:22:19.274 INFO:teuthology.orchestra.run.smithi142.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 596 kB/s | 45 kB 00:00 2024-04-26T21:22:19.324 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T21:22:19.355 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-26T21:22:19.401 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-26T21:22:19.419 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-26T21:22:19.509 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T21:22:19.549 INFO:teuthology.orchestra.run.smithi142.stdout:(102/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 24 MB/s | 16 MB 00:00 2024-04-26T21:22:19.559 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T21:22:19.638 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T21:22:19.782 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T21:22:19.838 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T21:22:19.891 INFO:teuthology.orchestra.run.smithi142.stdout:(103/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.7 MB/s | 548 kB 00:00 2024-04-26T21:22:19.916 INFO:teuthology.orchestra.run.smithi142.stdout:(104/152): lua-devel-5.4.4-4.el9.x86_64.rpm 32 kB/s | 22 kB 00:00 2024-04-26T21:22:20.008 INFO:teuthology.orchestra.run.smithi142.stdout:(105/152): gperftools-libs-2.9.1-2.el9.x86_64.r 2.6 MB/s | 309 kB 00:00 2024-04-26T21:22:20.108 INFO:teuthology.orchestra.run.smithi142.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 101 kB/s | 19 kB 00:00 2024-04-26T21:22:20.158 INFO:teuthology.orchestra.run.smithi142.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 520 kB/s | 26 kB 00:00 2024-04-26T21:22:20.218 INFO:teuthology.orchestra.run.smithi142.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 832 kB/s | 49 kB 00:00 2024-04-26T21:22:20.276 INFO:teuthology.orchestra.run.smithi142.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-04-26T21:22:20.367 INFO:teuthology.orchestra.run.smithi142.stdout:(110/152): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-04-26T21:22:20.399 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T21:22:20.468 INFO:teuthology.orchestra.run.smithi142.stdout:(111/152): libarrow-9.0.0-11.el9.x86_64.rpm 9.6 MB/s | 4.4 MB 00:00 2024-04-26T21:22:20.485 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T21:22:20.519 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T21:22:20.562 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T21:22:20.568 INFO:teuthology.orchestra.run.smithi142.stdout:(112/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.1 MB/s | 839 kB 00:00 2024-04-26T21:22:20.593 INFO:teuthology.orchestra.run.smithi142.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 4.3 MB/s | 548 kB 00:00 2024-04-26T21:22:20.600 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-26T21:22:20.629 INFO:teuthology.orchestra.run.smithi142.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 744 kB/s | 43 kB 00:00 2024-04-26T21:22:20.642 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T21:22:20.669 INFO:teuthology.orchestra.run.smithi142.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 428 kB/s | 32 kB 00:00 2024-04-26T21:22:20.694 INFO:teuthology.orchestra.run.smithi142.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 224 kB/s | 15 kB 00:00 2024-04-26T21:22:20.708 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-04-26T21:22:20.744 INFO:teuthology.orchestra.run.smithi142.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-04-26T21:22:20.787 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T21:22:20.803 INFO:teuthology.orchestra.run.smithi142.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 3.2 MB/s | 358 kB 00:00 2024-04-26T21:22:20.811 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T21:22:20.828 INFO:teuthology.orchestra.run.smithi142.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.5 MB/s | 303 kB 00:00 2024-04-26T21:22:20.835 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T21:22:20.862 INFO:teuthology.orchestra.run.smithi142.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 604 kB/s | 35 kB 00:00 2024-04-26T21:22:20.869 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T21:22:20.896 INFO:teuthology.orchestra.run.smithi142.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 3.0 MB/s | 206 kB 00:00 2024-04-26T21:22:20.925 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T21:22:20.937 INFO:teuthology.orchestra.run.smithi142.stdout:(122/152): protobuf-compiler-3.14.0-13.el9.x86_ 519 kB/s | 863 kB 00:01 2024-04-26T21:22:20.962 INFO:teuthology.orchestra.run.smithi142.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-04-26T21:22:20.974 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T21:22:21.001 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T21:22:21.021 INFO:teuthology.orchestra.run.smithi142.stdout:(124/152): python3-jaraco-classes-3.2.1-5.el9.n 302 kB/s | 18 kB 00:00 2024-04-26T21:22:21.039 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T21:22:21.073 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T21:22:21.080 INFO:teuthology.orchestra.run.smithi142.stdout:(125/152): python3-jaraco-collections-3.0.0-8.e 397 kB/s | 23 kB 00:00 2024-04-26T21:22:21.114 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T21:22:21.121 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T21:22:21.122 INFO:teuthology.orchestra.run.smithi142.stdout:(126/152): python3-jaraco-8.2.1-3.el9.noarch.rp 58 kB/s | 11 kB 00:00 2024-04-26T21:22:21.139 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T21:22:21.147 INFO:teuthology.orchestra.run.smithi142.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 292 kB/s | 19 kB 00:00 2024-04-26T21:22:21.156 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T21:22:21.184 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T21:22:21.199 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T21:22:21.205 INFO:teuthology.orchestra.run.smithi142.stdout:(128/152): python3-grpcio-1.46.7-10.el9.x86_64. 5.9 MB/s | 2.0 MB 00:00 2024-04-26T21:22:21.231 INFO:teuthology.orchestra.run.smithi142.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 180 kB/s | 20 kB 00:00 2024-04-26T21:22:21.241 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T21:22:21.264 INFO:teuthology.orchestra.run.smithi142.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 789 kB/s | 46 kB 00:00 2024-04-26T21:22:21.298 INFO:teuthology.orchestra.run.smithi142.stdout:(131/152): python3-kubernetes-26.1.0-2.el9.noar 6.8 MB/s | 1.0 MB 00:00 2024-04-26T21:22:21.332 INFO:teuthology.orchestra.run.smithi142.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 862 kB/s | 58 kB 00:00 2024-04-26T21:22:21.357 INFO:teuthology.orchestra.run.smithi142.stdout:(133/152): python3-more-itertools-8.12.0-2.el9. 628 kB/s | 79 kB 00:00 2024-04-26T21:22:21.382 INFO:teuthology.orchestra.run.smithi142.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-04-26T21:22:21.386 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T21:22:21.407 INFO:teuthology.orchestra.run.smithi142.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 217 kB/s | 16 kB 00:00 2024-04-26T21:22:21.430 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T21:22:21.449 INFO:teuthology.orchestra.run.smithi142.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 422 kB/s | 28 kB 00:00 2024-04-26T21:22:21.460 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T21:22:21.466 INFO:teuthology.orchestra.run.smithi142.stdout:(137/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 824 kB/s | 90 kB 00:00 2024-04-26T21:22:21.491 INFO:teuthology.orchestra.run.smithi142.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 368 kB/s | 31 kB 00:00 2024-04-26T21:22:21.502 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T21:22:21.525 INFO:teuthology.orchestra.run.smithi142.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-26T21:22:21.535 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T21:22:21.550 INFO:teuthology.orchestra.run.smithi142.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 611 kB/s | 36 kB 00:00 2024-04-26T21:22:21.576 INFO:teuthology.orchestra.run.smithi142.stdout:(141/152): python3-rsa-4.9-2.el9.noarch.rpm 542 kB/s | 59 kB 00:00 2024-04-26T21:22:21.601 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T21:22:21.601 INFO:teuthology.orchestra.run.smithi142.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 677 kB/s | 51 kB 00:00 2024-04-26T21:22:21.635 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T21:22:21.635 INFO:teuthology.orchestra.run.smithi142.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-04-26T21:22:21.656 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T21:22:21.660 INFO:teuthology.orchestra.run.smithi142.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 1.0 MB/s | 90 kB 00:00 2024-04-26T21:22:21.675 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T21:22:21.693 INFO:teuthology.orchestra.run.smithi142.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 4.5 MB/s | 427 kB 00:00 2024-04-26T21:22:21.708 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T21:22:21.719 INFO:teuthology.orchestra.run.smithi142.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 238 kB/s | 20 kB 00:00 2024-04-26T21:22:21.733 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T21:22:21.744 INFO:teuthology.orchestra.run.smithi142.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-04-26T21:22:21.781 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T21:22:21.808 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T21:22:21.811 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T21:22:21.844 INFO:teuthology.orchestra.run.smithi142.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-26T21:22:21.900 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T21:22:21.942 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T21:22:21.959 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T21:22:21.982 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T21:22:22.010 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T21:22:22.044 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T21:22:22.124 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T21:22:22.163 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T21:22:22.176 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T21:22:22.204 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T21:22:22.309 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T21:22:22.338 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T21:22:22.378 INFO:teuthology.orchestra.run.smithi142.stdout:(149/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 5.0 MB/s | 3.1 MB 00:00 2024-04-26T21:22:22.384 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T21:22:22.406 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T21:22:22.409 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T21:22:22.428 INFO:teuthology.orchestra.run.smithi142.stdout:(150/152): librados2-19.0.0-3306.gbc612ffb.el9. 4.8 MB/s | 3.4 MB 00:00 2024-04-26T21:22:22.432 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T21:22:22.446 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T21:22:22.481 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T21:22:22.531 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T21:22:22.574 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T21:22:22.607 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T21:22:22.624 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T21:22:22.656 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T21:22:22.703 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T21:22:22.746 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T21:22:22.768 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T21:22:22.784 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T21:22:22.833 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T21:22:22.864 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T21:22:22.901 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T21:22:22.944 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T21:22:22.949 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T21:22:23.026 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T21:22:23.044 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T21:22:23.073 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T21:22:23.117 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T21:22:23.149 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T21:22:23.177 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T21:22:23.248 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T21:22:23.301 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T21:22:23.330 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T21:22:23.365 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T21:22:23.443 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T21:22:23.484 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-26T21:22:23.528 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-26T21:22:23.556 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-26T21:22:23.629 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T21:22:23.680 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T21:22:23.753 INFO:teuthology.orchestra.run.smithi142.stdout:(151/152): openssl-3.2.1-1.el9.x86_64.rpm 724 kB/s | 1.3 MB 00:01 2024-04-26T21:22:23.758 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T21:22:23.813 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T21:22:23.865 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T21:22:23.899 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T21:22:23.902 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T21:22:23.966 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T21:22:24.001 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T21:22:24.058 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T21:22:24.148 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T21:22:24.184 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T21:22:24.369 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T21:22:24.398 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T21:22:24.478 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T21:22:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T21:22:24.630 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T21:22:24.665 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T21:22:24.708 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T21:22:24.746 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-26T21:22:24.788 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T21:22:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T21:22:25.032 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T21:22:25.088 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T21:22:25.163 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T21:22:25.165 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T21:22:25.208 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T21:22:25.244 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T21:22:25.283 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T21:22:25.291 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T21:22:25.338 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T21:22:25.379 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T21:22:25.421 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T21:22:25.566 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T21:22:25.569 INFO:teuthology.orchestra.run.smithi142.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 768 kB/s | 2.4 MB 00:03 2024-04-26T21:22:25.577 INFO:teuthology.orchestra.run.smithi142.stdout:-------------------------------------------------------------------------------- 2024-04-26T21:22:25.577 INFO:teuthology.orchestra.run.smithi142.stdout:Total 12 MB/s | 222 MB 00:18 2024-04-26T21:22:25.609 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T21:22:25.640 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T21:22:25.682 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T21:22:25.780 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T21:22:25.815 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T21:22:25.855 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T21:22:25.880 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T21:22:26.019 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T21:22:26.213 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T21:22:26.346 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T21:22:26.387 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T21:22:26.415 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T21:22:26.449 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T21:22:26.489 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T21:22:26.529 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T21:22:26.558 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T21:22:26.593 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T21:22:26.595 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T21:22:26.626 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T21:22:26.668 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T21:22:26.676 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T21:22:26.693 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction check 2024-04-26T21:22:26.694 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T21:22:26.705 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T21:22:26.738 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T21:22:26.785 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T21:22:26.834 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T21:22:26.837 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction check succeeded. 2024-04-26T21:22:26.837 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction test 2024-04-26T21:22:26.864 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T21:22:26.912 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T21:22:27.023 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T21:22:27.066 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T21:22:27.108 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T21:22:27.109 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T21:22:27.200 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T21:22:27.230 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-26T21:22:27.266 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T21:22:27.771 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T21:22:27.798 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T21:22:27.866 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T21:22:27.878 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T21:22:27.887 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-26T21:22:27.930 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T21:22:27.980 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T21:22:28.083 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T21:22:28.139 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T21:22:28.229 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T21:22:28.274 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T21:22:28.459 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T21:22:28.496 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T21:22:28.576 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T21:22:28.664 INFO:teuthology.orchestra.run.smithi142.stdout:Transaction test succeeded. 2024-04-26T21:22:28.665 INFO:teuthology.orchestra.run.smithi142.stdout:Running transaction 2024-04-26T21:22:29.260 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T21:22:29.383 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T21:22:30.322 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T21:22:30.351 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T21:22:30.692 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T21:22:30.776 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T21:22:30.805 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T21:22:30.837 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T21:22:30.921 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T21:22:30.962 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T21:22:31.041 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T21:22:31.041 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T21:22:31.041 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T21:22:31.041 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:22:31.072 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T21:22:31.165 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T21:22:31.200 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T21:22:31.283 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T21:22:31.313 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-26T21:22:31.349 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T21:22:31.854 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T21:22:31.878 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T21:22:31.958 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T21:22:31.988 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-26T21:22:34.311 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T21:22:34.343 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T21:22:34.573 INFO:teuthology.orchestra.run.smithi142.stdout: Preparing : 1/1 2024-04-26T21:22:34.648 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T21:22:34.686 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T21:22:34.740 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T21:22:34.774 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T21:22:34.799 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T21:22:34.842 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T21:22:34.912 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T21:22:34.963 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T21:22:35.048 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T21:22:35.048 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T21:22:35.048 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T21:22:35.048 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T21:22:35.081 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T21:22:35.216 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T21:22:35.267 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T21:22:35.388 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T21:22:35.410 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T21:22:35.501 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T21:22:35.538 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T21:22:35.686 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T21:22:35.744 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T21:22:35.773 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T21:22:35.858 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T21:22:35.881 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T21:22:35.927 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T21:22:35.948 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T21:22:35.973 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T21:22:36.173 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T21:22:36.206 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T21:22:36.313 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T21:22:36.410 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T21:22:36.438 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T21:22:36.472 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T21:22:36.538 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T21:22:36.571 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T21:22:36.626 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T21:22:36.658 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T21:22:36.686 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T21:22:36.755 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T21:22:36.808 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T21:22:36.839 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T21:22:36.882 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T21:22:36.958 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T21:22:36.995 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-26T21:22:37.027 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-26T21:22:37.056 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-26T21:22:37.128 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T21:22:37.170 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T21:22:37.246 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T21:22:37.379 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T21:22:37.432 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T21:22:37.973 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T21:22:38.052 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T21:22:38.081 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T21:22:38.124 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T21:22:38.153 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-26T21:22:38.195 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T21:22:38.357 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T21:22:38.414 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T21:22:38.469 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T21:22:38.542 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T21:22:38.573 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T21:22:38.618 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T21:22:38.665 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T21:22:38.700 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T21:22:38.743 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T21:22:38.785 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T21:22:38.921 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T21:22:38.965 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T21:22:38.996 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T21:22:39.037 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T21:22:39.131 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T21:22:39.162 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T21:22:39.202 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T21:22:39.228 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T21:22:39.252 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T21:22:39.327 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T21:22:39.453 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T21:22:39.484 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T21:22:39.504 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T21:22:39.537 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T21:22:39.577 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T21:22:39.609 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T21:22:39.638 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T21:22:39.673 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T21:22:39.706 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T21:22:39.747 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T21:22:39.782 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T21:22:39.870 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T21:22:39.920 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T21:22:39.952 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T21:22:39.998 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T21:22:40.097 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T21:22:40.144 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T21:22:40.959 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T21:22:41.008 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T21:22:41.042 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T21:22:41.140 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T21:22:41.191 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T21:22:41.278 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T21:22:41.310 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T21:22:41.479 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T21:22:41.508 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T21:22:41.585 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T21:22:42.226 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T21:22:42.363 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T21:22:42.657 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T21:22:42.658 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /sys 2024-04-26T21:22:42.658 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /proc 2024-04-26T21:22:42.658 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /mnt 2024-04-26T21:22:42.658 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /var/tmp 2024-04-26T21:22:42.658 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /home 2024-04-26T21:22:42.658 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /root 2024-04-26T21:22:42.658 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /tmp 2024-04-26T21:22:42.658 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:22:42.840 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T21:22:42.856 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T21:22:42.856 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:22:42.856 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T21:22:42.856 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:22:43.574 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T21:22:43.652 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T21:22:43.679 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T21:22:43.713 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T21:22:44.026 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T21:22:44.067 INFO:teuthology.orchestra.run.smithi142.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T21:22:44.157 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T21:22:44.189 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-26T21:22:44.224 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T21:22:44.392 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T21:22:44.419 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T21:22:44.419 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:22:44.419 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T21:22:44.419 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T21:22:44.419 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T21:22:44.420 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:22:44.652 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T21:22:44.676 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T21:22:44.676 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:22:44.676 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T21:22:44.676 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T21:22:44.677 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T21:22:44.677 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:22:44.705 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T21:22:44.728 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T21:22:44.800 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T21:22:44.830 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-26T21:22:45.145 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T21:22:45.169 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T21:22:45.169 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:22:45.169 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T21:22:45.169 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T21:22:45.169 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T21:22:45.169 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:22:45.275 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T21:22:45.333 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T21:22:45.397 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T21:22:45.591 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T21:22:45.638 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T21:22:45.719 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T21:22:45.809 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T21:22:45.960 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T21:22:46.004 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T21:22:46.036 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T21:22:46.063 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T21:22:46.484 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T21:22:46.512 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T21:22:46.544 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-26T21:22:46.617 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-26T21:22:46.643 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T21:22:47.138 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T21:22:47.139 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /sys 2024-04-26T21:22:47.139 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /proc 2024-04-26T21:22:47.139 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /mnt 2024-04-26T21:22:47.139 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /var/tmp 2024-04-26T21:22:47.139 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /home 2024-04-26T21:22:47.139 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /root 2024-04-26T21:22:47.139 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /tmp 2024-04-26T21:22:47.139 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T21:22:47.150 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T21:22:47.192 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T21:22:47.311 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T21:22:47.328 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T21:22:47.328 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:22:47.328 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T21:22:47.328 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T21:22:47.734 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T21:22:47.738 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T21:22:47.769 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T21:22:47.770 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T21:22:47.862 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T21:22:47.863 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T21:22:47.863 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T21:22:47.863 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-04-26T21:22:47.896 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T21:22:48.784 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T21:22:48.812 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T21:22:48.856 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T21:22:48.884 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T21:22:48.884 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:22:48.884 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T21:22:48.884 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T21:22:48.884 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T21:22:48.884 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T21:22:48.921 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T21:22:49.097 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-26T21:22:49.115 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T21:22:49.130 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T21:22:49.139 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T21:22:49.139 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:22:49.139 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T21:22:49.139 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T21:22:49.139 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T21:22:49.139 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T21:22:49.154 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T21:22:49.154 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:22:49.155 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T21:22:49.155 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T21:22:49.155 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T21:22:49.155 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:22:49.214 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T21:22:49.226 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T21:22:49.327 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-26T21:22:49.381 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-26T21:22:49.607 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T21:22:49.632 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T21:22:49.632 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:22:49.632 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T21:22:49.632 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T21:22:49.632 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T21:22:49.632 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T21:22:49.747 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T21:22:49.805 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T21:22:49.868 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T21:22:50.073 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T21:22:50.134 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T21:22:50.214 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T21:22:50.305 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T21:22:50.354 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T21:22:50.384 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T21:22:50.408 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T21:22:50.408 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:22:50.408 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T21:22:50.408 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T21:22:50.408 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T21:22:50.409 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:22:50.448 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T21:22:50.451 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T21:22:50.474 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T21:22:50.474 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:22:50.474 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T21:22:50.475 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:22:50.492 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T21:22:50.525 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T21:22:50.526 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T21:22:50.551 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T21:22:50.551 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:22:50.551 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T21:22:50.552 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T21:22:50.552 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T21:22:50.552 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:22:50.553 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T21:22:50.885 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T21:22:50.910 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T21:22:50.910 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:22:50.910 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T21:22:50.911 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T21:22:50.911 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T21:22:50.911 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:22:50.981 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T21:22:51.009 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T21:22:51.041 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-26T21:22:51.114 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-26T21:22:51.140 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T21:22:52.233 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T21:22:52.275 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T21:22:53.289 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T21:22:53.317 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T21:22:53.428 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T21:22:53.661 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-26T21:22:53.694 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T21:22:53.719 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T21:22:53.719 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:22:53.720 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T21:22:53.720 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T21:22:53.720 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T21:22:53.720 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T21:22:53.770 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T21:22:53.782 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T21:22:53.887 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-26T21:22:53.936 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-26T21:22:54.910 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T21:22:54.956 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T21:22:54.980 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T21:22:54.981 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:22:54.981 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T21:22:54.981 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T21:22:54.981 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T21:22:54.981 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T21:22:55.023 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T21:22:55.046 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T21:22:55.046 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:22:55.047 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T21:22:55.047 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T21:22:55.098 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T21:22:55.123 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T21:22:55.123 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:22:55.123 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T21:22:55.123 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T21:22:55.123 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T21:22:55.123 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T21:22:55.441 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T21:22:55.465 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T21:22:55.465 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:22:55.465 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T21:22:55.465 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T21:22:55.465 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T21:22:55.465 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T21:22:55.757 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-26T21:22:55.798 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-26T21:22:55.839 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-26T21:22:55.870 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T21:22:55.897 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T21:22:56.095 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-26T21:22:56.143 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T21:22:56.173 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-26T21:22:56.734 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T21:22:56.778 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T21:22:56.799 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T21:22:57.706 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-26T21:22:57.791 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T21:22:57.834 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T21:22:57.917 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T21:22:57.945 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T21:22:57.945 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T21:22:58.029 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T21:22:58.071 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T21:22:58.071 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T21:22:58.159 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T21:22:58.231 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T21:22:59.389 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T21:22:59.389 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /sys 2024-04-26T21:22:59.389 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /proc 2024-04-26T21:22:59.389 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /mnt 2024-04-26T21:22:59.389 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /var/tmp 2024-04-26T21:22:59.389 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /home 2024-04-26T21:22:59.389 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /root 2024-04-26T21:22:59.389 INFO:teuthology.orchestra.run.smithi142.stdout:skipping the directory /tmp 2024-04-26T21:22:59.389 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-04-26T21:22:59.558 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T21:22:59.570 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T21:22:59.570 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:22:59.571 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T21:22:59.571 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-04-26T21:23:00.321 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-26T21:23:00.362 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-26T21:23:00.433 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-26T21:23:00.468 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T21:23:00.495 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T21:23:00.698 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-26T21:23:00.741 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T21:23:00.771 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-26T21:23:01.058 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T21:23:01.083 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T21:23:01.084 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:23:01.084 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T21:23:01.084 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T21:23:01.084 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T21:23:01.084 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-04-26T21:23:01.318 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T21:23:01.341 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T21:23:01.341 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:23:01.341 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T21:23:01.341 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T21:23:01.341 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T21:23:01.341 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-04-26T21:23:01.390 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T21:23:01.427 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T21:23:01.448 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T21:23:01.786 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T21:23:01.809 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T21:23:01.809 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:23:01.810 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T21:23:01.810 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T21:23:01.810 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T21:23:01.810 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-04-26T21:23:01.923 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T21:23:01.972 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T21:23:02.028 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T21:23:02.211 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T21:23:02.260 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T21:23:02.330 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T21:23:02.366 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-26T21:23:02.418 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T21:23:02.449 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T21:23:02.491 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T21:23:02.558 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T21:23:02.576 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T21:23:02.602 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T21:23:02.612 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T21:23:02.612 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T21:23:02.634 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T21:23:02.662 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T21:23:02.696 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T21:23:02.731 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T21:23:02.731 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T21:23:02.825 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T21:23:02.900 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T21:23:03.139 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T21:23:03.168 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T21:23:03.201 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-26T21:23:03.295 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-26T21:23:03.325 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T21:23:04.066 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T21:23:04.066 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-26T21:23:04.066 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-26T21:23:04.066 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-26T21:23:04.066 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-26T21:23:04.066 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-26T21:23:04.067 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-26T21:23:04.067 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-26T21:23:04.067 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T21:23:04.067 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T21:23:04.067 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T21:23:04.067 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-26T21:23:04.067 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-26T21:23:04.067 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T21:23:04.068 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-26T21:23:04.068 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-26T21:23:04.068 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-26T21:23:04.069 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-26T21:23:04.069 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-26T21:23:04.069 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-26T21:23:04.069 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-26T21:23:04.069 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-26T21:23:04.070 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-26T21:23:04.070 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-26T21:23:04.070 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-26T21:23:04.070 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-26T21:23:04.070 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-26T21:23:04.070 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-26T21:23:04.070 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-26T21:23:04.070 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-26T21:23:04.070 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-26T21:23:04.070 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-26T21:23:04.070 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-26T21:23:04.070 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-26T21:23:04.073 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-26T21:23:04.074 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-26T21:23:04.074 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-26T21:23:04.074 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-26T21:23:04.074 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-26T21:23:04.074 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-26T21:23:04.074 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T21:23:04.075 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T21:23:04.075 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T21:23:04.075 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T21:23:04.075 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T21:23:04.075 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T21:23:04.076 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T21:23:04.076 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T21:23:04.076 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T21:23:04.076 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T21:23:04.076 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T21:23:04.076 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T21:23:04.077 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T21:23:04.077 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T21:23:04.077 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T21:23:04.077 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T21:23:04.077 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T21:23:04.078 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T21:23:04.078 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T21:23:04.078 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T21:23:04.078 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T21:23:04.078 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T21:23:04.078 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T21:23:04.079 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T21:23:04.079 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T21:23:04.079 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T21:23:04.080 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T21:23:04.080 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T21:23:04.080 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T21:23:04.080 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T21:23:04.081 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T21:23:04.081 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T21:23:04.081 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T21:23:04.081 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T21:23:04.081 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T21:23:04.081 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T21:23:04.082 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T21:23:04.082 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T21:23:04.082 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T21:23:04.082 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T21:23:04.082 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T21:23:04.082 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T21:23:04.082 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T21:23:04.082 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T21:23:04.082 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T21:23:04.082 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T21:23:04.082 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T21:23:04.082 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T21:23:04.082 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T21:23:04.083 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T21:23:04.083 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T21:23:04.083 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T21:23:04.083 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T21:23:04.083 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T21:23:04.083 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T21:23:04.083 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T21:23:04.083 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T21:23:04.083 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T21:23:04.083 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T21:23:04.083 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T21:23:04.083 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T21:23:04.083 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T21:23:04.083 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T21:23:04.084 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T21:23:04.084 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T21:23:04.084 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T21:23:04.084 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T21:23:04.084 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T21:23:04.084 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T21:23:04.084 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T21:23:04.084 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T21:23:04.084 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T21:23:04.084 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T21:23:04.084 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T21:23:04.084 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T21:23:04.084 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T21:23:04.084 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T21:23:04.085 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T21:23:04.085 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T21:23:04.085 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T21:23:04.085 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T21:23:04.085 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T21:23:04.085 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T21:23:04.085 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T21:23:04.085 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T21:23:04.086 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T21:23:04.086 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T21:23:04.086 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T21:23:04.086 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T21:23:04.086 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T21:23:04.086 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T21:23:04.086 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T21:23:04.086 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T21:23:04.086 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T21:23:04.086 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T21:23:04.086 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T21:23:04.087 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T21:23:04.087 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T21:23:04.087 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T21:23:04.087 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T21:23:04.087 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T21:23:04.087 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T21:23:04.087 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T21:23:04.087 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T21:23:04.087 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T21:23:04.087 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T21:23:04.087 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T21:23:04.087 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T21:23:04.087 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T21:23:04.087 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-26T21:23:04.088 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T21:23:04.088 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-26T21:23:04.088 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T21:23:04.088 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T21:23:04.088 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T21:23:04.088 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T21:23:04.366 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T21:23:04.410 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T21:23:05.373 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T21:23:05.401 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T21:23:05.502 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T21:23:05.662 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-26T21:23:05.687 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T21:23:05.710 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T21:23:05.710 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:23:05.710 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T21:23:05.710 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T21:23:05.710 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T21:23:05.710 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-04-26T21:23:05.746 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T21:23:05.757 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T21:23:05.846 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-26T21:23:05.902 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-26T21:23:06.210 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T21:23:06.211 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:23:06.211 INFO:teuthology.orchestra.run.smithi046.stdout:Upgraded: 2024-04-26T21:23:06.211 INFO:teuthology.orchestra.run.smithi046.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.211 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.211 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T21:23:06.211 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T21:23:06.211 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-04-26T21:23:06.211 INFO:teuthology.orchestra.run.smithi046.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T21:23:06.211 INFO:teuthology.orchestra.run.smithi046.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T21:23:06.211 INFO:teuthology.orchestra.run.smithi046.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T21:23:06.211 INFO:teuthology.orchestra.run.smithi046.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T21:23:06.211 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.211 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.212 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.212 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.212 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:06.212 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.212 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.212 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.212 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:06.212 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:06.212 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:06.212 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:06.212 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:06.212 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.212 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.213 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:06.213 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.213 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.213 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.213 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:06.213 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:06.213 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.213 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.213 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:06.213 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T21:23:06.213 INFO:teuthology.orchestra.run.smithi046.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T21:23:06.213 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T21:23:06.213 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T21:23:06.214 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T21:23:06.214 INFO:teuthology.orchestra.run.smithi046.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T21:23:06.214 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T21:23:06.214 INFO:teuthology.orchestra.run.smithi046.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T21:23:06.214 INFO:teuthology.orchestra.run.smithi046.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T21:23:06.214 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T21:23:06.214 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T21:23:06.214 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.214 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.214 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.214 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T21:23:06.214 INFO:teuthology.orchestra.run.smithi046.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T21:23:06.214 INFO:teuthology.orchestra.run.smithi046.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T21:23:06.215 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T21:23:06.215 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T21:23:06.215 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.215 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.215 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T21:23:06.215 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.215 INFO:teuthology.orchestra.run.smithi046.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T21:23:06.215 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T21:23:06.215 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T21:23:06.215 INFO:teuthology.orchestra.run.smithi046.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T21:23:06.216 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T21:23:06.216 INFO:teuthology.orchestra.run.smithi046.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T21:23:06.216 INFO:teuthology.orchestra.run.smithi046.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T21:23:06.216 INFO:teuthology.orchestra.run.smithi046.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T21:23:06.216 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T21:23:06.216 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T21:23:06.216 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T21:23:06.216 INFO:teuthology.orchestra.run.smithi046.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T21:23:06.216 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T21:23:06.216 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T21:23:06.216 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T21:23:06.216 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T21:23:06.217 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T21:23:06.217 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T21:23:06.217 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T21:23:06.217 INFO:teuthology.orchestra.run.smithi046.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T21:23:06.217 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T21:23:06.217 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T21:23:06.217 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T21:23:06.217 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.217 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.217 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.217 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T21:23:06.217 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T21:23:06.217 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T21:23:06.217 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T21:23:06.217 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T21:23:06.218 INFO:teuthology.orchestra.run.smithi046.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T21:23:06.218 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T21:23:06.218 INFO:teuthology.orchestra.run.smithi046.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T21:23:06.218 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T21:23:06.218 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T21:23:06.218 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T21:23:06.218 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T21:23:06.218 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T21:23:06.218 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T21:23:06.218 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T21:23:06.218 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T21:23:06.218 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T21:23:06.218 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T21:23:06.218 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T21:23:06.219 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T21:23:06.219 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T21:23:06.219 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T21:23:06.219 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T21:23:06.219 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T21:23:06.219 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T21:23:06.219 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T21:23:06.219 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T21:23:06.219 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T21:23:06.219 INFO:teuthology.orchestra.run.smithi046.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T21:23:06.219 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T21:23:06.219 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T21:23:06.219 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T21:23:06.219 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T21:23:06.220 INFO:teuthology.orchestra.run.smithi046.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T21:23:06.220 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T21:23:06.220 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T21:23:06.220 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T21:23:06.220 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T21:23:06.220 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T21:23:06.220 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T21:23:06.220 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.220 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.220 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T21:23:06.220 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T21:23:06.220 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T21:23:06.220 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.220 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T21:23:06.221 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T21:23:06.221 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T21:23:06.221 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T21:23:06.221 INFO:teuthology.orchestra.run.smithi046.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T21:23:06.221 INFO:teuthology.orchestra.run.smithi046.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T21:23:06.221 INFO:teuthology.orchestra.run.smithi046.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T21:23:06.221 INFO:teuthology.orchestra.run.smithi046.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T21:23:06.221 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T21:23:06.221 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T21:23:06.221 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T21:23:06.221 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T21:23:06.221 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T21:23:06.221 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T21:23:06.221 INFO:teuthology.orchestra.run.smithi046.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T21:23:06.222 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.222 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.222 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:06.222 INFO:teuthology.orchestra.run.smithi046.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T21:23:06.222 INFO:teuthology.orchestra.run.smithi046.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T21:23:06.222 INFO:teuthology.orchestra.run.smithi046.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T21:23:06.222 INFO:teuthology.orchestra.run.smithi046.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T21:23:06.222 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T21:23:06.222 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T21:23:06.222 INFO:teuthology.orchestra.run.smithi046.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T21:23:06.223 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:23:06.223 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-04-26T21:23:06.565 DEBUG:teuthology.parallel:result is None 2024-04-26T21:23:06.828 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T21:23:06.865 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T21:23:06.888 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T21:23:06.888 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:23:06.888 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T21:23:06.888 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T21:23:06.888 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T21:23:06.888 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-04-26T21:23:06.933 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T21:23:06.954 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T21:23:06.954 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:23:06.954 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T21:23:06.954 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-04-26T21:23:06.999 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T21:23:07.022 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T21:23:07.022 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:23:07.022 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T21:23:07.022 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T21:23:07.022 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T21:23:07.022 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-04-26T21:23:07.325 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T21:23:07.348 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T21:23:07.348 INFO:teuthology.orchestra.run.smithi142.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:23:07.348 INFO:teuthology.orchestra.run.smithi142.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T21:23:07.348 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T21:23:07.348 INFO:teuthology.orchestra.run.smithi142.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T21:23:07.348 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-04-26T21:23:08.621 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T21:23:08.621 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-26T21:23:08.621 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-26T21:23:08.621 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-26T21:23:08.621 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-26T21:23:08.622 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-26T21:23:08.622 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-26T21:23:08.622 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-26T21:23:08.622 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T21:23:08.622 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T21:23:08.622 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T21:23:08.622 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-26T21:23:08.622 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-26T21:23:08.622 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T21:23:08.622 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-26T21:23:08.622 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-26T21:23:08.622 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-26T21:23:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-26T21:23:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-26T21:23:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-26T21:23:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-26T21:23:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-26T21:23:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-26T21:23:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-26T21:23:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-26T21:23:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-26T21:23:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-26T21:23:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-26T21:23:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-26T21:23:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-26T21:23:08.623 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-26T21:23:08.624 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-26T21:23:08.624 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-26T21:23:08.624 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-26T21:23:08.624 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-26T21:23:08.624 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-26T21:23:08.624 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-26T21:23:08.624 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-26T21:23:08.624 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-26T21:23:08.624 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-26T21:23:08.624 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T21:23:08.624 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T21:23:08.624 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T21:23:08.624 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T21:23:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T21:23:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T21:23:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T21:23:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T21:23:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T21:23:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T21:23:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T21:23:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T21:23:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T21:23:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T21:23:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T21:23:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T21:23:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T21:23:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T21:23:08.625 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T21:23:08.626 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T21:23:08.626 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T21:23:08.626 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T21:23:08.626 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T21:23:08.626 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T21:23:08.626 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T21:23:08.626 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T21:23:08.626 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T21:23:08.626 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T21:23:08.626 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T21:23:08.626 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T21:23:08.626 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T21:23:08.627 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T21:23:08.627 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T21:23:08.627 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T21:23:08.627 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T21:23:08.627 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T21:23:08.627 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T21:23:08.627 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T21:23:08.627 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T21:23:08.627 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T21:23:08.627 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T21:23:08.627 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T21:23:08.627 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T21:23:08.627 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T21:23:08.627 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T21:23:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T21:23:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T21:23:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T21:23:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T21:23:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T21:23:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T21:23:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T21:23:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T21:23:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T21:23:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T21:23:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T21:23:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T21:23:08.628 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T21:23:08.629 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T21:23:08.629 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T21:23:08.629 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T21:23:08.629 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T21:23:08.629 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T21:23:08.629 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T21:23:08.629 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T21:23:08.629 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T21:23:08.629 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T21:23:08.629 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T21:23:08.629 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T21:23:08.629 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T21:23:08.629 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T21:23:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T21:23:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T21:23:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T21:23:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T21:23:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T21:23:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T21:23:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T21:23:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T21:23:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T21:23:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T21:23:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T21:23:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T21:23:08.630 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T21:23:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T21:23:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T21:23:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T21:23:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T21:23:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T21:23:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T21:23:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T21:23:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T21:23:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T21:23:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T21:23:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T21:23:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T21:23:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T21:23:08.631 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T21:23:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T21:23:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T21:23:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T21:23:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T21:23:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T21:23:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T21:23:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T21:23:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T21:23:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T21:23:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T21:23:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T21:23:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-26T21:23:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T21:23:08.632 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-26T21:23:08.633 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T21:23:08.633 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T21:23:08.633 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T21:23:08.633 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T21:23:10.680 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T21:23:10.681 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T21:23:10.681 INFO:teuthology.orchestra.run.smithi050.stdout:Upgraded: 2024-04-26T21:23:10.681 INFO:teuthology.orchestra.run.smithi050.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.681 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.681 INFO:teuthology.orchestra.run.smithi050.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T21:23:10.681 INFO:teuthology.orchestra.run.smithi050.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T21:23:10.681 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-04-26T21:23:10.681 INFO:teuthology.orchestra.run.smithi050.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T21:23:10.681 INFO:teuthology.orchestra.run.smithi050.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T21:23:10.681 INFO:teuthology.orchestra.run.smithi050.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T21:23:10.681 INFO:teuthology.orchestra.run.smithi050.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T21:23:10.682 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.682 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.682 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.682 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.682 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:10.682 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.682 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.682 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.682 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:10.682 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:10.682 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:10.682 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:10.683 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:10.683 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.683 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.683 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:10.683 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.683 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.683 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.683 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:10.683 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:10.683 INFO:teuthology.orchestra.run.smithi050.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.683 INFO:teuthology.orchestra.run.smithi050.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.683 INFO:teuthology.orchestra.run.smithi050.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:10.684 INFO:teuthology.orchestra.run.smithi050.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T21:23:10.684 INFO:teuthology.orchestra.run.smithi050.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T21:23:10.684 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T21:23:10.684 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T21:23:10.684 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T21:23:10.684 INFO:teuthology.orchestra.run.smithi050.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T21:23:10.684 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T21:23:10.684 INFO:teuthology.orchestra.run.smithi050.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T21:23:10.684 INFO:teuthology.orchestra.run.smithi050.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T21:23:10.684 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T21:23:10.684 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T21:23:10.684 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.684 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.685 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.685 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T21:23:10.685 INFO:teuthology.orchestra.run.smithi050.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T21:23:10.685 INFO:teuthology.orchestra.run.smithi050.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T21:23:10.685 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T21:23:10.685 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T21:23:10.685 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.685 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.685 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T21:23:10.685 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.685 INFO:teuthology.orchestra.run.smithi050.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T21:23:10.685 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T21:23:10.685 INFO:teuthology.orchestra.run.smithi050.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T21:23:10.686 INFO:teuthology.orchestra.run.smithi050.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T21:23:10.686 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T21:23:10.686 INFO:teuthology.orchestra.run.smithi050.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T21:23:10.686 INFO:teuthology.orchestra.run.smithi050.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T21:23:10.686 INFO:teuthology.orchestra.run.smithi050.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T21:23:10.686 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T21:23:10.686 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T21:23:10.686 INFO:teuthology.orchestra.run.smithi050.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T21:23:10.686 INFO:teuthology.orchestra.run.smithi050.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T21:23:10.686 INFO:teuthology.orchestra.run.smithi050.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T21:23:10.686 INFO:teuthology.orchestra.run.smithi050.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T21:23:10.686 INFO:teuthology.orchestra.run.smithi050.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T21:23:10.686 INFO:teuthology.orchestra.run.smithi050.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T21:23:10.687 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T21:23:10.687 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T21:23:10.687 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T21:23:10.687 INFO:teuthology.orchestra.run.smithi050.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T21:23:10.687 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T21:23:10.687 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T21:23:10.687 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T21:23:10.687 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.687 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.687 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.687 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T21:23:10.687 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T21:23:10.687 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T21:23:10.688 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T21:23:10.688 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T21:23:10.688 INFO:teuthology.orchestra.run.smithi050.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T21:23:10.688 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T21:23:10.688 INFO:teuthology.orchestra.run.smithi050.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T21:23:10.688 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T21:23:10.688 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T21:23:10.688 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T21:23:10.688 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T21:23:10.688 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T21:23:10.688 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T21:23:10.688 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T21:23:10.688 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T21:23:10.688 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T21:23:10.689 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T21:23:10.689 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T21:23:10.689 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T21:23:10.689 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T21:23:10.689 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T21:23:10.689 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T21:23:10.689 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T21:23:10.689 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T21:23:10.689 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T21:23:10.689 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T21:23:10.689 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T21:23:10.689 INFO:teuthology.orchestra.run.smithi050.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T21:23:10.689 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T21:23:10.689 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T21:23:10.690 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T21:23:10.690 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T21:23:10.690 INFO:teuthology.orchestra.run.smithi050.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T21:23:10.690 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T21:23:10.690 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T21:23:10.690 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T21:23:10.690 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T21:23:10.690 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T21:23:10.690 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T21:23:10.691 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.691 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.691 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T21:23:10.691 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T21:23:10.691 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T21:23:10.691 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.691 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T21:23:10.691 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T21:23:10.691 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T21:23:10.691 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T21:23:10.691 INFO:teuthology.orchestra.run.smithi050.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T21:23:10.691 INFO:teuthology.orchestra.run.smithi050.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T21:23:10.691 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T21:23:10.691 INFO:teuthology.orchestra.run.smithi050.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T21:23:10.692 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T21:23:10.692 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T21:23:10.692 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T21:23:10.692 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T21:23:10.692 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T21:23:10.692 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T21:23:10.692 INFO:teuthology.orchestra.run.smithi050.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T21:23:10.692 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.692 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.692 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:10.692 INFO:teuthology.orchestra.run.smithi050.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T21:23:10.692 INFO:teuthology.orchestra.run.smithi050.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T21:23:10.692 INFO:teuthology.orchestra.run.smithi050.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T21:23:10.692 INFO:teuthology.orchestra.run.smithi050.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T21:23:10.693 INFO:teuthology.orchestra.run.smithi050.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T21:23:10.693 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T21:23:10.693 INFO:teuthology.orchestra.run.smithi050.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T21:23:10.693 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T21:23:10.693 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-04-26T21:23:11.104 DEBUG:teuthology.parallel:result is None 2024-04-26T21:23:11.945 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-26T21:23:11.987 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-26T21:23:12.027 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-26T21:23:12.059 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T21:23:12.087 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T21:23:12.268 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-26T21:23:12.307 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T21:23:12.337 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-26T21:23:12.900 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T21:23:12.934 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T21:23:12.954 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T21:23:13.782 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-26T21:23:13.872 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T21:23:13.912 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T21:23:13.993 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T21:23:14.025 INFO:teuthology.orchestra.run.smithi142.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T21:23:14.026 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T21:23:14.109 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T21:23:14.142 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T21:23:14.143 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T21:23:14.233 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T21:23:14.299 INFO:teuthology.orchestra.run.smithi142.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T21:23:20.095 INFO:teuthology.orchestra.run.smithi142.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T21:23:20.095 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-26T21:23:20.095 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-26T21:23:20.095 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-26T21:23:20.095 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-26T21:23:20.095 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-26T21:23:20.096 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-26T21:23:20.096 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-26T21:23:20.096 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T21:23:20.096 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T21:23:20.096 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T21:23:20.096 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-26T21:23:20.096 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-26T21:23:20.096 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T21:23:20.096 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-26T21:23:20.096 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-26T21:23:20.096 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-26T21:23:20.096 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-26T21:23:20.096 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-26T21:23:20.096 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-26T21:23:20.097 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-26T21:23:20.097 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-26T21:23:20.097 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-26T21:23:20.097 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-26T21:23:20.097 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-26T21:23:20.097 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-26T21:23:20.097 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-26T21:23:20.097 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-26T21:23:20.097 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-26T21:23:20.097 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-26T21:23:20.097 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-26T21:23:20.109 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-26T21:23:20.110 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-26T21:23:20.110 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-26T21:23:20.110 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-26T21:23:20.110 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-26T21:23:20.110 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-26T21:23:20.110 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-26T21:23:20.110 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-26T21:23:20.110 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-26T21:23:20.110 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T21:23:20.110 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T21:23:20.110 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T21:23:20.110 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T21:23:20.111 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T21:23:20.111 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T21:23:20.111 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T21:23:20.111 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T21:23:20.111 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T21:23:20.111 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T21:23:20.111 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T21:23:20.111 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T21:23:20.111 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T21:23:20.111 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T21:23:20.111 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T21:23:20.111 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T21:23:20.111 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T21:23:20.112 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T21:23:20.112 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T21:23:20.112 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T21:23:20.112 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T21:23:20.112 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T21:23:20.112 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T21:23:20.112 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T21:23:20.112 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T21:23:20.112 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T21:23:20.112 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T21:23:20.112 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T21:23:20.112 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T21:23:20.112 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T21:23:20.112 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T21:23:20.113 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T21:23:20.113 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T21:23:20.113 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T21:23:20.113 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T21:23:20.113 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T21:23:20.113 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T21:23:20.113 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T21:23:20.113 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T21:23:20.113 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T21:23:20.114 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T21:23:20.114 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T21:23:20.114 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T21:23:20.114 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T21:23:20.114 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T21:23:20.114 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T21:23:20.114 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T21:23:20.114 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T21:23:20.114 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T21:23:20.114 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T21:23:20.114 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T21:23:20.114 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T21:23:20.115 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T21:23:20.115 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T21:23:20.115 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T21:23:20.115 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T21:23:20.115 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T21:23:20.115 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T21:23:20.115 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T21:23:20.115 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T21:23:20.115 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T21:23:20.115 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T21:23:20.115 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T21:23:20.115 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T21:23:20.115 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T21:23:20.115 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T21:23:20.116 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T21:23:20.116 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T21:23:20.116 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T21:23:20.116 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T21:23:20.116 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T21:23:20.116 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T21:23:20.116 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T21:23:20.116 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T21:23:20.116 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T21:23:20.116 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T21:23:20.116 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T21:23:20.116 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T21:23:20.116 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T21:23:20.116 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T21:23:20.116 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T21:23:20.117 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T21:23:20.117 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T21:23:20.117 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T21:23:20.117 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T21:23:20.117 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T21:23:20.117 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T21:23:20.117 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T21:23:20.117 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T21:23:20.117 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T21:23:20.117 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T21:23:20.117 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T21:23:20.117 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T21:23:20.117 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T21:23:20.117 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T21:23:20.118 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T21:23:20.118 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T21:23:20.118 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T21:23:20.118 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T21:23:20.118 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T21:23:20.118 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T21:23:20.118 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T21:23:20.118 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T21:23:20.119 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T21:23:20.119 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T21:23:20.119 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T21:23:20.119 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T21:23:20.119 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T21:23:20.119 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T21:23:20.120 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-26T21:23:20.120 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T21:23:20.120 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-26T21:23:20.120 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T21:23:20.120 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T21:23:20.120 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T21:23:20.121 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T21:23:22.203 INFO:teuthology.orchestra.run.smithi142.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T21:23:22.204 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-04-26T21:23:22.204 INFO:teuthology.orchestra.run.smithi142.stdout:Upgraded: 2024-04-26T21:23:22.204 INFO:teuthology.orchestra.run.smithi142.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.204 INFO:teuthology.orchestra.run.smithi142.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.204 INFO:teuthology.orchestra.run.smithi142.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T21:23:22.204 INFO:teuthology.orchestra.run.smithi142.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T21:23:22.204 INFO:teuthology.orchestra.run.smithi142.stdout:Installed: 2024-04-26T21:23:22.204 INFO:teuthology.orchestra.run.smithi142.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T21:23:22.204 INFO:teuthology.orchestra.run.smithi142.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T21:23:22.204 INFO:teuthology.orchestra.run.smithi142.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T21:23:22.204 INFO:teuthology.orchestra.run.smithi142.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T21:23:22.204 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.205 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.205 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.205 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.205 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:22.205 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.205 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.205 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.205 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:22.205 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:22.205 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:22.205 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:22.205 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:23:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T21:23:22.206 INFO:teuthology.orchestra.run.smithi142.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T21:23:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T21:23:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T21:23:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T21:23:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T21:23:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T21:23:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T21:23:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T21:23:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T21:23:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T21:23:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.207 INFO:teuthology.orchestra.run.smithi142.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T21:23:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T21:23:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T21:23:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T21:23:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T21:23:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T21:23:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T21:23:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T21:23:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T21:23:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T21:23:22.208 INFO:teuthology.orchestra.run.smithi142.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T21:23:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T21:23:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T21:23:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T21:23:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T21:23:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T21:23:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T21:23:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T21:23:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T21:23:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T21:23:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T21:23:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T21:23:22.209 INFO:teuthology.orchestra.run.smithi142.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T21:23:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T21:23:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T21:23:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T21:23:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T21:23:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T21:23:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T21:23:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T21:23:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T21:23:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T21:23:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T21:23:22.210 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T21:23:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T21:23:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T21:23:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T21:23:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T21:23:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T21:23:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T21:23:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T21:23:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T21:23:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T21:23:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T21:23:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T21:23:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T21:23:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T21:23:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T21:23:22.211 INFO:teuthology.orchestra.run.smithi142.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T21:23:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T21:23:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T21:23:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T21:23:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T21:23:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T21:23:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T21:23:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T21:23:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T21:23:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T21:23:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T21:23:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T21:23:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T21:23:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T21:23:22.212 INFO:teuthology.orchestra.run.smithi142.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T21:23:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T21:23:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T21:23:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T21:23:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T21:23:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T21:23:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T21:23:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T21:23:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T21:23:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T21:23:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T21:23:22.213 INFO:teuthology.orchestra.run.smithi142.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T21:23:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T21:23:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T21:23:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T21:23:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T21:23:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T21:23:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T21:23:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T21:23:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T21:23:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T21:23:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T21:23:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T21:23:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T21:23:22.214 INFO:teuthology.orchestra.run.smithi142.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T21:23:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:23:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T21:23:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T21:23:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T21:23:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T21:23:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T21:23:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T21:23:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T21:23:22.215 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-04-26T21:23:22.216 INFO:teuthology.orchestra.run.smithi142.stdout:Complete! 2024-04-26T21:23:22.627 DEBUG:teuthology.parallel:result is None 2024-04-26T21:23:22.627 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T21:23:22.769 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T21:23:22.796 INFO:teuthology.orchestra.run.smithi046.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-26T21:23:22.796 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-26T21:23:22.797 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-26T21:23:22.798 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T21:23:22.951 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T21:23:22.977 INFO:teuthology.orchestra.run.smithi050.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-26T21:23:22.978 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-26T21:23:22.978 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-26T21:23:22.980 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T21:23:23.119 DEBUG:teuthology.orchestra.run.smithi142:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T21:23:23.141 INFO:teuthology.orchestra.run.smithi142.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-26T21:23:23.141 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-26T21:23:23.141 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-26T21:23:23.143 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-26T21:23:23.143 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-26T21:23:23.144 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T21:23:23.176 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T21:23:23.176 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T21:23:23.209 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-04-26T21:23:23.209 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T21:23:23.235 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-26T21:23:23.235 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-26T21:23:23.235 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T21:23:23.264 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T21:23:23.332 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T21:23:23.332 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T21:23:23.362 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T21:23:23.430 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-04-26T21:23:23.430 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T21:23:23.453 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T21:23:23.516 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-26T21:23:23.517 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-26T21:23:23.517 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T21:23:23.546 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T21:23:23.615 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T21:23:23.615 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T21:23:23.641 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T21:23:23.706 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-04-26T21:23:23.706 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T21:23:23.735 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T21:23:23.798 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-26T21:23:23.799 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-26T21:23:23.799 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T21:23:23.823 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T21:23:23.888 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T21:23:23.889 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T21:23:23.914 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T21:23:23.978 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-04-26T21:23:23.978 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T21:23:24.001 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T21:23:24.065 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-26T21:23:24.166 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-26T21:23:24.167 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T21:23:24.167 INFO:tasks.cephadm:Cluster fsid is 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:23:24.167 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-26T21:23:24.168 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.46', 'mon.b': '172.21.15.50', 'mon.c': '172.21.15.142'} 2024-04-26T21:23:24.168 INFO:tasks.cephadm:First mon is mon.a on smithi046 2024-04-26T21:23:24.168 INFO:tasks.cephadm:First mgr is x 2024-04-26T21:23:24.168 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-26T21:23:24.168 DEBUG:teuthology.orchestra.run.smithi046:> sudo hostname $(hostname -s) 2024-04-26T21:23:24.194 DEBUG:teuthology.orchestra.run.smithi050:> sudo hostname $(hostname -s) 2024-04-26T21:23:24.218 DEBUG:teuthology.orchestra.run.smithi142:> sudo hostname $(hostname -s) 2024-04-26T21:23:24.242 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-26T21:23:24.242 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T21:23:24.550 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78918/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3306-gbc612ffb', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3306.gbc612ffb'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-26 13:02:42.256491', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-rishabh-testing-20240426.111959', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-26T21:23:24.685 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-rishabh-testing-20240426.111959, sha1 bc612ffb9c1d5bc27536bc917a705262a36d4387 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T21:23:24.687 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm 2024-04-26T21:23:24.687 DEBUG:teuthology.orchestra.run.smithi046:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T21:23:25.130 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 26 21:23 /home/ubuntu/cephtest/cephadm 2024-04-26T21:23:25.131 DEBUG:teuthology.orchestra.run.smithi050:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T21:23:25.725 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 26 21:23 /home/ubuntu/cephtest/cephadm 2024-04-26T21:23:25.726 DEBUG:teuthology.orchestra.run.smithi142:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T21:23:26.178 INFO:teuthology.orchestra.run.smithi142.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 26 21:23 /home/ubuntu/cephtest/cephadm 2024-04-26T21:23:26.178 DEBUG:teuthology.orchestra.run.smithi046:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T21:23:26.198 DEBUG:teuthology.orchestra.run.smithi050:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T21:23:26.218 DEBUG:teuthology.orchestra.run.smithi142:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T21:23:26.253 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 on all hosts... 2024-04-26T21:23:26.253 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-26T21:23:26.256 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-26T21:23:26.261 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-26T21:23:27.764 INFO:teuthology.orchestra.run.smithi046.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T21:23:27.890 INFO:teuthology.orchestra.run.smithi142.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T21:23:27.920 INFO:teuthology.orchestra.run.smithi050.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T21:24:13.604 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-26T21:24:13.604 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T21:24:13.604 INFO:teuthology.orchestra.run.smithi050.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-26T21:24:13.604 INFO:teuthology.orchestra.run.smithi050.stdout: "repo_digests": [ 2024-04-26T21:24:13.604 INFO:teuthology.orchestra.run.smithi050.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-26T21:24:13.604 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T21:24:13.604 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-26T21:24:16.420 INFO:teuthology.orchestra.run.smithi142.stdout:{ 2024-04-26T21:24:16.420 INFO:teuthology.orchestra.run.smithi142.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T21:24:16.420 INFO:teuthology.orchestra.run.smithi142.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-26T21:24:16.421 INFO:teuthology.orchestra.run.smithi142.stdout: "repo_digests": [ 2024-04-26T21:24:16.421 INFO:teuthology.orchestra.run.smithi142.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-26T21:24:16.421 INFO:teuthology.orchestra.run.smithi142.stdout: ] 2024-04-26T21:24:16.421 INFO:teuthology.orchestra.run.smithi142.stdout:} 2024-04-26T21:24:21.338 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-26T21:24:21.339 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T21:24:21.339 INFO:teuthology.orchestra.run.smithi046.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-26T21:24:21.339 INFO:teuthology.orchestra.run.smithi046.stdout: "repo_digests": [ 2024-04-26T21:24:21.339 INFO:teuthology.orchestra.run.smithi046.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-26T21:24:21.339 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-04-26T21:24:21.339 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-26T21:24:21.362 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /etc/ceph 2024-04-26T21:24:21.400 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /etc/ceph 2024-04-26T21:24:21.432 DEBUG:teuthology.orchestra.run.smithi142:> sudo mkdir -p /etc/ceph 2024-04-26T21:24:21.463 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 777 /etc/ceph 2024-04-26T21:24:21.487 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 777 /etc/ceph 2024-04-26T21:24:21.515 DEBUG:teuthology.orchestra.run.smithi142:> sudo chmod 777 /etc/ceph 2024-04-26T21:24:21.542 INFO:tasks.cephadm:Writing seed config... 2024-04-26T21:24:21.543 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-26T21:24:21.543 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-26T21:24:21.543 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-26T21:24:21.543 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-26T21:24:21.543 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-26T21:24:21.543 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-26T21:24:21.543 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-26T21:24:21.543 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-26T21:24:21.543 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-26T21:24:21.544 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-26T21:24:21.544 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-26T21:24:21.544 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-26T21:24:21.544 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-26T21:24:21.544 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-26T21:24:21.544 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-26T21:24:21.544 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-26T21:24:21.544 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-26T21:24:21.544 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-04-26T21:24:21.544 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-26T21:24:21.544 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-26T21:24:21.544 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-26T21:24:21.544 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-26T21:24:21.544 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-26T21:24:21.544 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-26T21:24:21.545 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-26T21:24:21.545 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-26T21:24:21.545 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-26T21:24:21.545 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-26T21:24:21.545 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-26T21:24:21.545 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-26T21:24:21.545 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-26T21:24:21.545 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-26T21:24:21.545 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-26T21:24:21.545 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-26T21:24:21.545 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-26T21:24:21.545 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-26T21:24:21.545 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-26T21:24:21.545 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-26T21:24:21.545 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-26T21:24:21.545 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-26T21:24:21.546 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-26T21:24:21.546 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-26T21:24:21.546 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-26T21:24:21.546 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-26T21:24:21.546 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-26T21:24:21.546 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-04-26T21:24:21.547 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-26T21:24:21.547 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-26T21:24:21.562 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 36c7013a-0413-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-26T21:24:21.562 DEBUG:teuthology.orchestra.run.smithi046:mon.a> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mon.a.service 2024-04-26T21:24:21.604 DEBUG:teuthology.orchestra.run.smithi046:mgr.x> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mgr.x.service 2024-04-26T21:24:21.646 INFO:tasks.cephadm:Bootstrapping... 2024-04-26T21:24:21.647 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 -v bootstrap --fsid 36c7013a-0413-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.46 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-26T21:24:21.843 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-04-26T21:24:21.843 INFO:teuthology.orchestra.run.smithi046.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387', '-v', 'bootstrap', '--fsid', '36c7013a-0413-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.46', '--skip-admin-label'] 2024-04-26T21:24:21.843 INFO:teuthology.orchestra.run.smithi046.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-26T21:24:21.844 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying podman|docker is present... 2024-04-26T21:24:21.869 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-26T21:24:21.869 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying lvm2 is present... 2024-04-26T21:24:21.869 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying time synchronization is in place... 2024-04-26T21:24:21.876 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T21:24:21.877 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T21:24:21.882 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T21:24:21.883 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2024-04-26T21:24:21.889 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout enabled 2024-04-26T21:24:21.895 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout active 2024-04-26T21:24:21.895 INFO:teuthology.orchestra.run.smithi046.stdout:Unit chronyd.service is enabled and running 2024-04-26T21:24:21.895 INFO:teuthology.orchestra.run.smithi046.stdout:Repeating the final host check... 2024-04-26T21:24:21.919 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-26T21:24:21.919 INFO:teuthology.orchestra.run.smithi046.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-26T21:24:21.919 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl is present 2024-04-26T21:24:21.919 INFO:teuthology.orchestra.run.smithi046.stdout:lvcreate is present 2024-04-26T21:24:21.925 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T21:24:21.926 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T21:24:21.932 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T21:24:21.932 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2024-04-26T21:24:21.938 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout enabled 2024-04-26T21:24:21.944 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout active 2024-04-26T21:24:21.945 INFO:teuthology.orchestra.run.smithi046.stdout:Unit chronyd.service is enabled and running 2024-04-26T21:24:21.945 INFO:teuthology.orchestra.run.smithi046.stdout:Host looks OK 2024-04-26T21:24:21.945 INFO:teuthology.orchestra.run.smithi046.stdout:Cluster fsid: 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:24:21.945 INFO:teuthology.orchestra.run.smithi046.stdout:Acquiring lock 140614479314080 on /run/cephadm/36c7013a-0413-11ef-bc93-c7b262605968.lock 2024-04-26T21:24:21.945 INFO:teuthology.orchestra.run.smithi046.stdout:Lock 140614479314080 acquired on /run/cephadm/36c7013a-0413-11ef-bc93-c7b262605968.lock 2024-04-26T21:24:21.946 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying IP 172.21.15.46 port 3300 ... 2024-04-26T21:24:21.946 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying IP 172.21.15.46 port 6789 ... 2024-04-26T21:24:21.946 INFO:teuthology.orchestra.run.smithi046.stdout:Base mon IP(s) is [172.21.15.46:3300, 172.21.15.46:6789], mon addrv is [v2:172.21.15.46:3300,v1:172.21.15.46:6789] 2024-04-26T21:24:21.949 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.46 metric 100 2024-04-26T21:24:21.949 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.46 metric 100 2024-04-26T21:24:21.951 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-26T21:24:21.952 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-26T21:24:21.952 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-26T21:24:21.954 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-26T21:24:21.954 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-26T21:24:21.954 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T21:24:21.954 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-26T21:24:21.954 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-04-26T21:24:21.954 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T21:24:21.955 INFO:teuthology.orchestra.run.smithi046.stdout:Mon IP `172.21.15.46` is in CIDR network `172.21.0.0/20` 2024-04-26T21:24:21.955 INFO:teuthology.orchestra.run.smithi046.stdout:Mon IP `172.21.15.46` is in CIDR network `172.21.0.0/20` 2024-04-26T21:24:21.955 INFO:teuthology.orchestra.run.smithi046.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-26T21:24:21.955 INFO:teuthology.orchestra.run.smithi046.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-26T21:24:21.956 INFO:teuthology.orchestra.run.smithi046.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T21:24:22.450 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stdout 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 2024-04-26T21:24:22.450 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T21:24:22.450 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Getting image source signatures 2024-04-26T21:24:22.450 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Copying blob sha256:f340d36a00bce3feed0ad509aeafc9092c1c625bf553b8a613c380a79149b87e 2024-04-26T21:24:22.450 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-26T21:24:22.451 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Copying config sha256:55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 2024-04-26T21:24:22.451 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-26T21:24:23.369 INFO:teuthology.orchestra.run.smithi046.stdout:ceph: stdout ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev) 2024-04-26T21:24:23.370 INFO:teuthology.orchestra.run.smithi046.stdout:Ceph version: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev) 2024-04-26T21:24:23.370 INFO:teuthology.orchestra.run.smithi046.stdout:Extracting ceph user uid/gid from container image... 2024-04-26T21:24:24.465 INFO:teuthology.orchestra.run.smithi046.stdout:stat: stdout 167 167 2024-04-26T21:24:24.465 INFO:teuthology.orchestra.run.smithi046.stdout:Creating initial keys... 2024-04-26T21:24:25.345 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph-authtool: stdout AQCIGyxmPLy1LxAAdNFgLeW88N6hdjvyCwK88Q== 2024-04-26T21:24:26.193 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph-authtool: stdout AQCJGyxm83EcJxAA9KdKqs/NXL8G1myq72Bl8g== 2024-04-26T21:24:27.077 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph-authtool: stdout AQCKGyxmRoi3HhAA6o+LX9WorT19llO19ImoNw== 2024-04-26T21:24:27.077 INFO:teuthology.orchestra.run.smithi046.stdout:Creating initial monmap... 2024-04-26T21:24:28.042 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T21:24:28.042 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-26T21:24:28.042 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:24:28.042 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T21:24:28.042 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool for a [v2:172.21.15.46:3300,v1:172.21.15.46:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T21:24:28.043 INFO:teuthology.orchestra.run.smithi046.stdout:setting min_mon_release = quincy 2024-04-26T21:24:28.043 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: set fsid to 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:24:28.043 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T21:24:28.043 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:24:28.043 INFO:teuthology.orchestra.run.smithi046.stdout:Creating mon... 2024-04-26T21:24:28.974 INFO:teuthology.orchestra.run.smithi046.stdout:create mon.a on 2024-04-26T21:24:29.184 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-26T21:24:29.347 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-26T21:24:29.605 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-36c7013a-0413-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-36c7013a-0413-11ef-bc93-c7b262605968.target. 2024-04-26T21:24:29.605 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-36c7013a-0413-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-36c7013a-0413-11ef-bc93-c7b262605968.target. 2024-04-26T21:24:29.835 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-36c7013a-0413-11ef-bc93-c7b262605968@mon.a 2024-04-26T21:24:29.835 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Failed to reset failed state of unit ceph-36c7013a-0413-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-36c7013a-0413-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-26T21:24:30.005 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-36c7013a-0413-11ef-bc93-c7b262605968.target.wants/ceph-36c7013a-0413-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-36c7013a-0413-11ef-bc93-c7b262605968@.service. 2024-04-26T21:24:30.518 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:30 smithi046 ceph-mon[28821]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T21:24:30.518 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:30 smithi046 bash[28802]: d87830d383fac8e4e01782d926671187c7d4cb56b831d6d3675a0c018947ce5e 2024-04-26T21:24:30.531 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T21:24:30.532 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout disabled 2024-04-26T21:24:30.539 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T21:24:30.539 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2024-04-26T21:24:30.539 INFO:teuthology.orchestra.run.smithi046.stdout:firewalld.service is not enabled 2024-04-26T21:24:30.539 INFO:teuthology.orchestra.run.smithi046.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T21:24:30.539 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mon to start... 2024-04-26T21:24:30.539 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mon... 2024-04-26T21:24:30.767 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:30 smithi046 systemd[1]: Started Ceph mon.a for 36c7013a-0413-11ef-bc93-c7b262605968. 2024-04-26T21:24:31.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout cluster: 2024-04-26T21:24:31.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout id: 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:24:31.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-26T21:24:31.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-04-26T21:24:31.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout services: 2024-04-26T21:24:31.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.546856s) 2024-04-26T21:24:31.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-26T21:24:31.594 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-26T21:24:31.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-04-26T21:24:31.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout data: 2024-04-26T21:24:31.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-26T21:24:31.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-26T21:24:31.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-26T21:24:31.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout pgs: 2024-04-26T21:24:31.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-04-26T21:24:31.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.945+0000 7fee33136700 1 Processor -- start 2024-04-26T21:24:31.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.946+0000 7fee33136700 1 -- start start 2024-04-26T21:24:31.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.946+0000 7fee33136700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee2c108d90 0x7fee2c1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:31.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.946+0000 7fee33136700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fee2c109780 con 0x7fee2c108d90 2024-04-26T21:24:31.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.946+0000 7fee30ed2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee2c108d90 0x7fee2c1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:31.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.946+0000 7fee30ed2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee2c108d90 0x7fee2c1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42626/0 (socket says 172.21.15.46:42626) 2024-04-26T21:24:31.595 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.946+0000 7fee30ed2700 1 -- 172.21.15.46:0/2756286586 learned_addr learned my addr 172.21.15.46:0/2756286586 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:24:31.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.946+0000 7fee30ed2700 1 -- 172.21.15.46:0/2756286586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee2c109f90 con 0x7fee2c108d90 2024-04-26T21:24:31.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.947+0000 7fee30ed2700 1 --2- 172.21.15.46:0/2756286586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee2c108d90 0x7fee2c1091b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fee1c0098a0 tx=0x7fee1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcbb34fa5a3d13b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:31.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.947+0000 7fee2b7fe700 1 -- 172.21.15.46:0/2756286586 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee1c004160 con 0x7fee2c108d90 2024-04-26T21:24:31.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.947+0000 7fee2b7fe700 1 -- 172.21.15.46:0/2756286586 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee1c004300 con 0x7fee2c108d90 2024-04-26T21:24:31.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.947+0000 7fee2b7fe700 1 -- 172.21.15.46:0/2756286586 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee1c0036a0 con 0x7fee2c108d90 2024-04-26T21:24:31.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.947+0000 7fee33136700 1 -- 172.21.15.46:0/2756286586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee2c108d90 msgr2=0x7fee2c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:31.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.947+0000 7fee33136700 1 --2- 172.21.15.46:0/2756286586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee2c108d90 0x7fee2c1091b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fee1c0098a0 tx=0x7fee1c009bb0 comp rx=0 tx=0).stop 2024-04-26T21:24:31.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.947+0000 7fee33136700 1 -- 172.21.15.46:0/2756286586 shutdown_connections 2024-04-26T21:24:31.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.947+0000 7fee33136700 1 --2- 172.21.15.46:0/2756286586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee2c108d90 0x7fee2c1091b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:31.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.947+0000 7fee33136700 1 -- 172.21.15.46:0/2756286586 >> 172.21.15.46:0/2756286586 conn(0x7fee2c104310 msgr2=0x7fee2c106770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:24:31.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.947+0000 7fee33136700 1 -- 172.21.15.46:0/2756286586 shutdown_connections 2024-04-26T21:24:31.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.947+0000 7fee33136700 1 -- 172.21.15.46:0/2756286586 wait complete. 2024-04-26T21:24:31.596 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.948+0000 7fee33136700 1 Processor -- start 2024-04-26T21:24:31.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.948+0000 7fee33136700 1 -- start start 2024-04-26T21:24:31.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.948+0000 7fee33136700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee2c108d90 0x7fee2c19c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:31.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.948+0000 7fee33136700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fee2c10a4c0 con 0x7fee2c108d90 2024-04-26T21:24:31.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.948+0000 7fee30ed2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee2c108d90 0x7fee2c19c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:31.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.948+0000 7fee30ed2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee2c108d90 0x7fee2c19c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42632/0 (socket says 172.21.15.46:42632) 2024-04-26T21:24:31.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.948+0000 7fee30ed2700 1 -- 172.21.15.46:0/1309024845 learned_addr learned my addr 172.21.15.46:0/1309024845 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:24:31.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.949+0000 7fee30ed2700 1 -- 172.21.15.46:0/1309024845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee2c19c9f0 con 0x7fee2c108d90 2024-04-26T21:24:31.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.949+0000 7fee30ed2700 1 --2- 172.21.15.46:0/1309024845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee2c108d90 0x7fee2c19c4b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fee1c000c00 tx=0x7fee1c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:31.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.949+0000 7fee29ffb700 1 -- 172.21.15.46:0/1309024845 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee1c003db0 con 0x7fee2c108d90 2024-04-26T21:24:31.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.949+0000 7fee29ffb700 1 -- 172.21.15.46:0/1309024845 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee1c011630 con 0x7fee2c108d90 2024-04-26T21:24:31.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.949+0000 7fee33136700 1 -- 172.21.15.46:0/1309024845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee2c19cc80 con 0x7fee2c108d90 2024-04-26T21:24:31.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.949+0000 7fee29ffb700 1 -- 172.21.15.46:0/1309024845 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee1c011950 con 0x7fee2c108d90 2024-04-26T21:24:31.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.949+0000 7fee33136700 1 -- 172.21.15.46:0/1309024845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee2c19f970 con 0x7fee2c108d90 2024-04-26T21:24:31.597 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.949+0000 7fee29ffb700 1 -- 172.21.15.46:0/1309024845 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fee1c01a460 con 0x7fee2c108d90 2024-04-26T21:24:31.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.949+0000 7fee29ffb700 1 -- 172.21.15.46:0/1309024845 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fee1c01baf0 con 0x7fee2c108d90 2024-04-26T21:24:31.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.950+0000 7fee33136700 1 -- 172.21.15.46:0/1309024845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee2c064a80 con 0x7fee2c108d90 2024-04-26T21:24:31.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:30.953+0000 7fee29ffb700 1 -- 172.21.15.46:0/1309024845 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fee1c031020 con 0x7fee2c108d90 2024-04-26T21:24:31.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:31.026+0000 7fee33136700 1 -- 172.21.15.46:0/1309024845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fee2c02e6e0 con 0x7fee2c108d90 2024-04-26T21:24:31.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:31.026+0000 7fee29ffb700 1 -- 172.21.15.46:0/1309024845 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7fee1c004260 con 0x7fee2c108d90 2024-04-26T21:24:31.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:31.027+0000 7fee33136700 1 -- 172.21.15.46:0/1309024845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee2c108d90 msgr2=0x7fee2c19c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:31.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:31.027+0000 7fee33136700 1 --2- 172.21.15.46:0/1309024845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee2c108d90 0x7fee2c19c4b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fee1c000c00 tx=0x7fee1c004750 comp rx=0 tx=0).stop 2024-04-26T21:24:31.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:31.027+0000 7fee33136700 1 -- 172.21.15.46:0/1309024845 shutdown_connections 2024-04-26T21:24:31.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:31.027+0000 7fee33136700 1 --2- 172.21.15.46:0/1309024845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee2c108d90 0x7fee2c19c4b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:31.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:31.027+0000 7fee33136700 1 -- 172.21.15.46:0/1309024845 >> 172.21.15.46:0/1309024845 conn(0x7fee2c104310 msgr2=0x7fee2c193090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:24:31.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:31.027+0000 7fee33136700 1 -- 172.21.15.46:0/1309024845 shutdown_connections 2024-04-26T21:24:31.598 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:31.027+0000 7fee33136700 1 -- 172.21.15.46:0/1309024845 wait complete. 2024-04-26T21:24:31.598 INFO:teuthology.orchestra.run.smithi046.stdout:mon is available 2024-04-26T21:24:31.598 INFO:teuthology.orchestra.run.smithi046.stdout:Assimilating anything we can from ceph.conf... 2024-04-26T21:24:31.703 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:31 smithi046 ceph-mon[28821]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T21:24:31.703 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:31 smithi046 ceph-mon[28821]: monmap epoch 1 2024-04-26T21:24:31.703 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:31 smithi046 ceph-mon[28821]: fsid 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:24:31.703 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:31 smithi046 ceph-mon[28821]: last_changed 2024-04-26T21:24:27.425916+0000 2024-04-26T21:24:31.703 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:31 smithi046 ceph-mon[28821]: created 2024-04-26T21:24:27.425916+0000 2024-04-26T21:24:31.703 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:31 smithi046 ceph-mon[28821]: min_mon_release 19 (squid) 2024-04-26T21:24:31.703 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:31 smithi046 ceph-mon[28821]: election_strategy: 1 2024-04-26T21:24:31.703 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:31 smithi046 ceph-mon[28821]: 0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-04-26T21:24:31.704 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:31 smithi046 ceph-mon[28821]: fsmap 2024-04-26T21:24:31.704 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:31 smithi046 ceph-mon[28821]: osdmap e1: 0 total, 0 up, 0 in 2024-04-26T21:24:31.704 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:31 smithi046 ceph-mon[28821]: mgrmap e1: no daemons active 2024-04-26T21:24:31.704 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:31 smithi046 ceph-mon[28821]: from='client.? 172.21.15.46:0/1309024845' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-26T21:24:32.636 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-04-26T21:24:32.636 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T21:24:32.636 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout fsid = 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:24:32.636 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-26T21:24:32.637 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.46:3300,v1:172.21.15.46:6789] 2024-04-26T21:24:32.637 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T21:24:32.637 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T21:24:32.637 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T21:24:32.637 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T21:24:32.637 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-04-26T21:24:32.637 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T21:24:32.637 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T21:24:32.637 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-04-26T21:24:32.637 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T21:24:32.637 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-26T21:24:32.637 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-26T21:24:32.637 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-26T21:24:32.637 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T21:24:32.638 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T21:24:32.638 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.024+0000 7f731232b700 1 Processor -- start 2024-04-26T21:24:32.638 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.024+0000 7f731232b700 1 -- start start 2024-04-26T21:24:32.638 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.024+0000 7f731232b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f730c107c90 0x7f730c1080b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:32.638 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.024+0000 7f731232b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f730c108680 con 0x7f730c107c90 2024-04-26T21:24:32.638 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.025+0000 7f730bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f730c107c90 0x7f730c1080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:32.638 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.025+0000 7f730bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f730c107c90 0x7f730c1080b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42646/0 (socket says 172.21.15.46:42646) 2024-04-26T21:24:32.638 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.025+0000 7f730bfff700 1 -- 172.21.15.46:0/1129041192 learned_addr learned my addr 172.21.15.46:0/1129041192 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:24:32.638 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.025+0000 7f730bfff700 1 -- 172.21.15.46:0/1129041192 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f730c108e90 con 0x7f730c107c90 2024-04-26T21:24:32.638 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.025+0000 7f730bfff700 1 --2- 172.21.15.46:0/1129041192 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f730c107c90 0x7f730c1080b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f72fc0098a0 tx=0x7f72fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fc7b9a063ba32bd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:32.638 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.025+0000 7f730affd700 1 -- 172.21.15.46:0/1129041192 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72fc004160 con 0x7f730c107c90 2024-04-26T21:24:32.638 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.025+0000 7f730affd700 1 -- 172.21.15.46:0/1129041192 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72fc004300 con 0x7f730c107c90 2024-04-26T21:24:32.638 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.025+0000 7f730affd700 1 -- 172.21.15.46:0/1129041192 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72fc0036a0 con 0x7f730c107c90 2024-04-26T21:24:32.638 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.026+0000 7f731232b700 1 -- 172.21.15.46:0/1129041192 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f730c107c90 msgr2=0x7f730c1080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:32.639 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.026+0000 7f731232b700 1 --2- 172.21.15.46:0/1129041192 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f730c107c90 0x7f730c1080b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f72fc0098a0 tx=0x7f72fc009bb0 comp rx=0 tx=0).stop 2024-04-26T21:24:32.639 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.026+0000 7f731232b700 1 -- 172.21.15.46:0/1129041192 shutdown_connections 2024-04-26T21:24:32.639 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.026+0000 7f731232b700 1 --2- 172.21.15.46:0/1129041192 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f730c107c90 0x7f730c1080b0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:32.639 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.026+0000 7f731232b700 1 -- 172.21.15.46:0/1129041192 >> 172.21.15.46:0/1129041192 conn(0x7f730c1031f0 msgr2=0x7f730c105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:24:32.639 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.026+0000 7f731232b700 1 -- 172.21.15.46:0/1129041192 shutdown_connections 2024-04-26T21:24:32.639 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.026+0000 7f731232b700 1 -- 172.21.15.46:0/1129041192 wait complete. 2024-04-26T21:24:32.639 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.027+0000 7f731232b700 1 Processor -- start 2024-04-26T21:24:32.639 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.027+0000 7f731232b700 1 -- start start 2024-04-26T21:24:32.639 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.027+0000 7f731232b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f730c107c90 0x7f730c19c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:32.639 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.027+0000 7f731232b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f730c1093c0 con 0x7f730c107c90 2024-04-26T21:24:32.639 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.027+0000 7f730bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f730c107c90 0x7f730c19c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:32.639 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.027+0000 7f730bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f730c107c90 0x7f730c19c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42662/0 (socket says 172.21.15.46:42662) 2024-04-26T21:24:32.639 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.027+0000 7f730bfff700 1 -- 172.21.15.46:0/4025668051 learned_addr learned my addr 172.21.15.46:0/4025668051 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:24:32.639 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.027+0000 7f730bfff700 1 -- 172.21.15.46:0/4025668051 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f730c19ca00 con 0x7f730c107c90 2024-04-26T21:24:32.640 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.028+0000 7f730bfff700 1 --2- 172.21.15.46:0/4025668051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f730c107c90 0x7f730c19c4c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f72fc000c00 tx=0x7f72fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:32.640 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.028+0000 7f73097fa700 1 -- 172.21.15.46:0/4025668051 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72fc003db0 con 0x7f730c107c90 2024-04-26T21:24:32.640 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.028+0000 7f73097fa700 1 -- 172.21.15.46:0/4025668051 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72fc011630 con 0x7f730c107c90 2024-04-26T21:24:32.640 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.028+0000 7f731232b700 1 -- 172.21.15.46:0/4025668051 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f730c19cc90 con 0x7f730c107c90 2024-04-26T21:24:32.640 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.028+0000 7f73097fa700 1 -- 172.21.15.46:0/4025668051 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72fc011950 con 0x7f730c107c90 2024-04-26T21:24:32.640 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.028+0000 7f731232b700 1 -- 172.21.15.46:0/4025668051 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f730c19f980 con 0x7f730c107c90 2024-04-26T21:24:32.640 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.028+0000 7f73097fa700 1 -- 172.21.15.46:0/4025668051 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f72fc01a460 con 0x7f730c107c90 2024-04-26T21:24:32.640 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.028+0000 7f73097fa700 1 -- 172.21.15.46:0/4025668051 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f72fc01b780 con 0x7f730c107c90 2024-04-26T21:24:32.640 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.029+0000 7f731232b700 1 -- 172.21.15.46:0/4025668051 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f730c064a80 con 0x7f730c107c90 2024-04-26T21:24:32.641 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.031+0000 7f73097fa700 1 -- 172.21.15.46:0/4025668051 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f72fc031020 con 0x7f730c107c90 2024-04-26T21:24:32.641 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.101+0000 7f731232b700 1 -- 172.21.15.46:0/4025668051 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f730c02e790 con 0x7f730c107c90 2024-04-26T21:24:32.641 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.104+0000 7f73097fa700 1 -- 172.21.15.46:0/4025668051 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+465 (secure 0 0 0) 0x7f72fc01b9d0 con 0x7f730c107c90 2024-04-26T21:24:32.641 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.105+0000 7f73097fa700 1 -- 172.21.15.46:0/4025668051 <== mon.0 v2:172.21.15.46:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f72fc01ad20 con 0x7f730c107c90 2024-04-26T21:24:32.641 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.105+0000 7f731232b700 1 -- 172.21.15.46:0/4025668051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f730c107c90 msgr2=0x7f730c19c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:32.641 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.105+0000 7f731232b700 1 --2- 172.21.15.46:0/4025668051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f730c107c90 0x7f730c19c4c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f72fc000c00 tx=0x7f72fc004750 comp rx=0 tx=0).stop 2024-04-26T21:24:32.641 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.106+0000 7f731232b700 1 -- 172.21.15.46:0/4025668051 shutdown_connections 2024-04-26T21:24:32.641 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.106+0000 7f731232b700 1 --2- 172.21.15.46:0/4025668051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f730c107c90 0x7f730c19c4c0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:32.641 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.106+0000 7f731232b700 1 -- 172.21.15.46:0/4025668051 >> 172.21.15.46:0/4025668051 conn(0x7f730c1031f0 msgr2=0x7f730c193050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:24:32.642 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.106+0000 7f731232b700 1 -- 172.21.15.46:0/4025668051 shutdown_connections 2024-04-26T21:24:32.642 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:32.106+0000 7f731232b700 1 -- 172.21.15.46:0/4025668051 wait complete. 2024-04-26T21:24:32.642 INFO:teuthology.orchestra.run.smithi046.stdout:Generating new minimal ceph.conf... 2024-04-26T21:24:32.736 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:32 smithi046 ceph-mon[28821]: from='client.? 172.21.15.46:0/4025668051' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T21:24:32.736 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:32 smithi046 ceph-mon[28821]: from='client.? 172.21.15.46:0/4025668051' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-26T21:24:33.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.023+0000 7f0552001700 1 Processor -- start 2024-04-26T21:24:33.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.023+0000 7f0552001700 1 -- start start 2024-04-26T21:24:33.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.024+0000 7f0552001700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f054c108db0 0x7f054c1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:33.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.024+0000 7f0552001700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f054c1097a0 con 0x7f054c108db0 2024-04-26T21:24:33.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.024+0000 7f054b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f054c108db0 0x7f054c1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:33.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.024+0000 7f054b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f054c108db0 0x7f054c1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42678/0 (socket says 172.21.15.46:42678) 2024-04-26T21:24:33.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.024+0000 7f054b7fe700 1 -- 172.21.15.46:0/2287231766 learned_addr learned my addr 172.21.15.46:0/2287231766 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:24:33.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.024+0000 7f054b7fe700 1 -- 172.21.15.46:0/2287231766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f054c109f60 con 0x7f054c108db0 2024-04-26T21:24:33.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.024+0000 7f054b7fe700 1 --2- 172.21.15.46:0/2287231766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f054c108db0 0x7f054c1091d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0544009e80 tx=0x7f054400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c6208025ea9eb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:33.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.024+0000 7f054a7fc700 1 -- 172.21.15.46:0/2287231766 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0544017070 con 0x7f054c108db0 2024-04-26T21:24:33.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.025+0000 7f054a7fc700 1 -- 172.21.15.46:0/2287231766 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f054400b8c0 con 0x7f054c108db0 2024-04-26T21:24:33.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.025+0000 7f054a7fc700 1 -- 172.21.15.46:0/2287231766 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0544004490 con 0x7f054c108db0 2024-04-26T21:24:33.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.025+0000 7f0552001700 1 -- 172.21.15.46:0/2287231766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f054c108db0 msgr2=0x7f054c1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:33.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.025+0000 7f0552001700 1 --2- 172.21.15.46:0/2287231766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f054c108db0 0x7f054c1091d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0544009e80 tx=0x7f054400b330 comp rx=0 tx=0).stop 2024-04-26T21:24:33.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.025+0000 7f0552001700 1 -- 172.21.15.46:0/2287231766 shutdown_connections 2024-04-26T21:24:33.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.025+0000 7f0552001700 1 --2- 172.21.15.46:0/2287231766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f054c108db0 0x7f054c1091d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:33.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.025+0000 7f0552001700 1 -- 172.21.15.46:0/2287231766 >> 172.21.15.46:0/2287231766 conn(0x7f054c104310 msgr2=0x7f054c106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:24:33.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.025+0000 7f0552001700 1 -- 172.21.15.46:0/2287231766 shutdown_connections 2024-04-26T21:24:33.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.025+0000 7f0552001700 1 -- 172.21.15.46:0/2287231766 wait complete. 2024-04-26T21:24:33.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.026+0000 7f0552001700 1 Processor -- start 2024-04-26T21:24:33.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.026+0000 7f0552001700 1 -- start start 2024-04-26T21:24:33.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.026+0000 7f0552001700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f054c108db0 0x7f054c07f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:33.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.026+0000 7f0552001700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f054c10d310 con 0x7f054c108db0 2024-04-26T21:24:33.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.026+0000 7f054b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f054c108db0 0x7f054c07f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:33.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.026+0000 7f054b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f054c108db0 0x7f054c07f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42694/0 (socket says 172.21.15.46:42694) 2024-04-26T21:24:33.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.026+0000 7f054b7fe700 1 -- 172.21.15.46:0/929296638 learned_addr learned my addr 172.21.15.46:0/929296638 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:24:33.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.027+0000 7f054b7fe700 1 -- 172.21.15.46:0/929296638 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f054c07f930 con 0x7f054c108db0 2024-04-26T21:24:33.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.027+0000 7f054b7fe700 1 --2- 172.21.15.46:0/929296638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f054c108db0 0x7f054c07f3f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0544004a60 tx=0x7f054400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:33.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.027+0000 7f0548ff9700 1 -- 172.21.15.46:0/929296638 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0544020020 con 0x7f054c108db0 2024-04-26T21:24:33.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.027+0000 7f0548ff9700 1 -- 172.21.15.46:0/929296638 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0544012430 con 0x7f054c108db0 2024-04-26T21:24:33.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.027+0000 7f0552001700 1 -- 172.21.15.46:0/929296638 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f054c07db20 con 0x7f054c108db0 2024-04-26T21:24:33.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.027+0000 7f0548ff9700 1 -- 172.21.15.46:0/929296638 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0544017040 con 0x7f054c108db0 2024-04-26T21:24:33.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.027+0000 7f0552001700 1 -- 172.21.15.46:0/929296638 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f054c07e060 con 0x7f054c108db0 2024-04-26T21:24:33.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.027+0000 7f0548ff9700 1 -- 172.21.15.46:0/929296638 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0544023020 con 0x7f054c108db0 2024-04-26T21:24:33.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.027+0000 7f0548ff9700 1 -- 172.21.15.46:0/929296638 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f054400f050 con 0x7f054c108db0 2024-04-26T21:24:33.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.028+0000 7f0552001700 1 -- 172.21.15.46:0/929296638 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f054c064a80 con 0x7f054c108db0 2024-04-26T21:24:33.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.030+0000 7f0548ff9700 1 -- 172.21.15.46:0/929296638 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f054402a020 con 0x7f054c108db0 2024-04-26T21:24:33.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.102+0000 7f0552001700 1 -- 172.21.15.46:0/929296638 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f054c07e2e0 con 0x7f054c108db0 2024-04-26T21:24:33.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.103+0000 7f0548ff9700 1 -- 172.21.15.46:0/929296638 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f054400b860 con 0x7f054c108db0 2024-04-26T21:24:33.670 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.103+0000 7f0552001700 1 -- 172.21.15.46:0/929296638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f054c108db0 msgr2=0x7f054c07f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:33.671 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.103+0000 7f0552001700 1 --2- 172.21.15.46:0/929296638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f054c108db0 0x7f054c07f3f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0544004a60 tx=0x7f054400bd50 comp rx=0 tx=0).stop 2024-04-26T21:24:33.671 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.103+0000 7f0552001700 1 -- 172.21.15.46:0/929296638 shutdown_connections 2024-04-26T21:24:33.671 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.103+0000 7f0552001700 1 --2- 172.21.15.46:0/929296638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f054c108db0 0x7f054c07f3f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:33.671 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.103+0000 7f0552001700 1 -- 172.21.15.46:0/929296638 >> 172.21.15.46:0/929296638 conn(0x7f054c104310 msgr2=0x7f054c104e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:24:33.671 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.104+0000 7f0552001700 1 -- 172.21.15.46:0/929296638 shutdown_connections 2024-04-26T21:24:33.671 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:33.104+0000 7f0552001700 1 -- 172.21.15.46:0/929296638 wait complete. 2024-04-26T21:24:33.671 INFO:teuthology.orchestra.run.smithi046.stdout:Restarting the monitor... 2024-04-26T21:24:33.679 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:33 smithi046 ceph-mon[28821]: from='client.? 172.21.15.46:0/929296638' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:34.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:33 smithi046 systemd[1]: Stopping Ceph mon.a for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:24:34.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:33 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mon-a[28817]: 2024-04-26T21:24:33.918+0000 7eff8bd6d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:24:34.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:33 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mon-a[28817]: 2024-04-26T21:24:33.918+0000 7eff8bd6d700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-26T21:24:34.427 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:34 smithi046 podman[29068]: 2024-04-26 21:24:34.09384049 +0000 UTC m=+0.267923321 container died d87830d383fac8e4e01782d926671187c7d4cb56b831d6d3675a0c018947ce5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-26T21:24:34.427 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:34 smithi046 podman[29068]: 2024-04-26 21:24:34.427255456 +0000 UTC m=+0.601338289 container cleanup d87830d383fac8e4e01782d926671187c7d4cb56b831d6d3675a0c018947ce5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-04-26T21:24:34.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:34 smithi046 bash[29068]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-mon-a 2024-04-26T21:24:34.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:34 smithi046 podman[29082]: 2024-04-26 21:24:34.594873397 +0000 UTC m=+0.501478300 container remove d87830d383fac8e4e01782d926671187c7d4cb56b831d6d3675a0c018947ce5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-26T21:24:35.151 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:34 smithi046 systemd[1]: ceph-36c7013a-0413-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-26T21:24:35.151 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:34 smithi046 systemd[1]: Stopped Ceph mon.a for 36c7013a-0413-11ef-bc93-c7b262605968. 2024-04-26T21:24:35.151 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:34 smithi046 systemd[1]: Starting Ceph mon.a for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:24:35.151 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 podman[29182]: 2024-04-26 21:24:35.062248615 +0000 UTC m=+0.095585596 container create fc596ea263bf65540b7a94fda1bb803cbcc1ec30cfa0880760818f6c09e7de32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-26T21:24:35.152 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 podman[29182]: 2024-04-26 21:24:34.984820653 +0000 UTC m=+0.018157598 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T21:24:35.310 INFO:teuthology.orchestra.run.smithi046.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-26T21:24:35.508 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 podman[29182]: 2024-04-26 21:24:35.239901424 +0000 UTC m=+0.273238381 container init fc596ea263bf65540b7a94fda1bb803cbcc1ec30cfa0880760818f6c09e7de32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-26T21:24:35.508 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 podman[29182]: 2024-04-26 21:24:35.242849344 +0000 UTC m=+0.276186286 container start fc596ea263bf65540b7a94fda1bb803cbcc1ec30cfa0880760818f6c09e7de32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2) 2024-04-26T21:24:35.508 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: set uid:gid to 167:167 (ceph:ceph) 2024-04-26T21:24:35.508 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev), process ceph-mon, pid 2 2024-04-26T21:24:35.508 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: pidfile_write: ignore empty --pid-file 2024-04-26T21:24:35.508 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: load: jerasure load: lrc 2024-04-26T21:24:35.508 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: RocksDB version: 7.9.2 2024-04-26T21:24:35.508 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Git sha 0 2024-04-26T21:24:35.508 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Compile date 2024-04-26 11:39:23 2024-04-26T21:24:35.509 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: DB SUMMARY 2024-04-26T21:24:35.509 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: DB Session ID: C2CG8KEX3FGPI3X23UNB 2024-04-26T21:24:35.509 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: CURRENT file: CURRENT 2024-04-26T21:24:35.509 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: IDENTITY file: IDENTITY 2024-04-26T21:24:35.509 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-26T21:24:35.509 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-26T21:24:35.509 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104650 ; 2024-04-26T21:24:35.509 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.error_if_exists: 0 2024-04-26T21:24:35.509 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.create_if_missing: 0 2024-04-26T21:24:35.509 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.paranoid_checks: 1 2024-04-26T21:24:35.509 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-26T21:24:35.509 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-26T21:24:35.509 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-26T21:24:35.509 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.env: 0x559607980cc0 2024-04-26T21:24:35.510 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.fs: PosixFileSystem 2024-04-26T21:24:35.510 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.info_log: 0x5596088b3d80 2024-04-26T21:24:35.510 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.max_file_opening_threads: 16 2024-04-26T21:24:35.511 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.statistics: (nil) 2024-04-26T21:24:35.511 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.use_fsync: 0 2024-04-26T21:24:35.511 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.max_log_file_size: 0 2024-04-26T21:24:35.511 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-26T21:24:35.511 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-26T21:24:35.511 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.keep_log_file_num: 1000 2024-04-26T21:24:35.511 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.recycle_log_file_num: 0 2024-04-26T21:24:35.511 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.allow_fallocate: 1 2024-04-26T21:24:35.511 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.allow_mmap_reads: 0 2024-04-26T21:24:35.511 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.allow_mmap_writes: 0 2024-04-26T21:24:35.511 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.use_direct_reads: 0 2024-04-26T21:24:35.511 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-26T21:24:35.511 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.create_missing_column_families: 0 2024-04-26T21:24:35.512 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.db_log_dir: 2024-04-26T21:24:35.512 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.wal_dir: 2024-04-26T21:24:35.512 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-26T21:24:35.512 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-26T21:24:35.512 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-26T21:24:35.512 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-26T21:24:35.512 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-26T21:24:35.512 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-26T21:24:35.512 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.advise_random_on_open: 1 2024-04-26T21:24:35.512 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.db_write_buffer_size: 0 2024-04-26T21:24:35.512 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.write_buffer_manager: 0x5596088850e0 2024-04-26T21:24:35.512 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-26T21:24:35.512 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-26T21:24:35.512 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-26T21:24:35.513 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.rate_limiter: (nil) 2024-04-26T21:24:35.513 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-26T21:24:35.513 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.wal_recovery_mode: 2 2024-04-26T21:24:35.513 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.enable_thread_tracking: 0 2024-04-26T21:24:35.513 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.enable_pipelined_write: 0 2024-04-26T21:24:35.513 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.unordered_write: 0 2024-04-26T21:24:35.513 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-26T21:24:35.513 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-26T21:24:35.513 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-26T21:24:35.513 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-26T21:24:35.513 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.row_cache: None 2024-04-26T21:24:35.513 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.wal_filter: None 2024-04-26T21:24:35.513 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-26T21:24:35.513 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.allow_ingest_behind: 0 2024-04-26T21:24:35.514 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.two_write_queues: 0 2024-04-26T21:24:35.514 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.manual_wal_flush: 0 2024-04-26T21:24:35.514 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.wal_compression: 0 2024-04-26T21:24:35.514 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.atomic_flush: 0 2024-04-26T21:24:35.514 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-26T21:24:35.514 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-26T21:24:35.514 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-26T21:24:35.514 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.log_readahead_size: 0 2024-04-26T21:24:35.514 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-26T21:24:35.514 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.best_efforts_recovery: 0 2024-04-26T21:24:35.514 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-26T21:24:35.514 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-26T21:24:35.514 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.allow_data_in_errors: 0 2024-04-26T21:24:35.515 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.db_host_id: __hostname__ 2024-04-26T21:24:35.515 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-26T21:24:35.515 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.max_background_jobs: 2 2024-04-26T21:24:35.515 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.max_background_compactions: -1 2024-04-26T21:24:35.515 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.max_subcompactions: 1 2024-04-26T21:24:35.515 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-26T21:24:35.516 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-26T21:24:35.516 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-26T21:24:35.516 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.max_total_wal_size: 0 2024-04-26T21:24:35.516 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-26T21:24:35.516 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-26T21:24:35.516 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-26T21:24:35.517 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-26T21:24:35.517 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.max_open_files: -1 2024-04-26T21:24:35.517 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.bytes_per_sync: 0 2024-04-26T21:24:35.517 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-26T21:24:35.517 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-26T21:24:35.517 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.compaction_readahead_size: 0 2024-04-26T21:24:35.517 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.max_background_flushes: -1 2024-04-26T21:24:35.518 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Compression algorithms supported: 2024-04-26T21:24:35.518 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-26T21:24:35.518 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: kZSTD supported: 0 2024-04-26T21:24:35.518 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: kXpressCompression supported: 0 2024-04-26T21:24:35.518 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: kLZ4HCCompression supported: 1 2024-04-26T21:24:35.518 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: kZlibCompression supported: 1 2024-04-26T21:24:35.519 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: kSnappyCompression supported: 1 2024-04-26T21:24:35.519 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: kLZ4Compression supported: 1 2024-04-26T21:24:35.519 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: kBZip2Compression supported: 0 2024-04-26T21:24:35.519 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-26T21:24:35.519 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-26T21:24:35.519 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-26T21:24:35.519 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-26T21:24:35.520 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-26T21:24:35.520 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.merge_operator: 2024-04-26T21:24:35.520 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.compaction_filter: None 2024-04-26T21:24:35.520 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.compaction_filter_factory: None 2024-04-26T21:24:35.520 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.sst_partitioner_factory: None 2024-04-26T21:24:35.520 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-26T21:24:35.521 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-26T21:24:35.521 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5596088b3f40) 2024-04-26T21:24:35.521 INFO:journalctl@ceph.mon.a.smithi046.stdout: cache_index_and_filter_blocks: 1 2024-04-26T21:24:35.521 INFO:journalctl@ceph.mon.a.smithi046.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-26T21:24:35.521 INFO:journalctl@ceph.mon.a.smithi046.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-26T21:24:35.521 INFO:journalctl@ceph.mon.a.smithi046.stdout: pin_top_level_index_and_filter: 1 2024-04-26T21:24:35.521 INFO:journalctl@ceph.mon.a.smithi046.stdout: index_type: 0 2024-04-26T21:24:35.522 INFO:journalctl@ceph.mon.a.smithi046.stdout: data_block_index_type: 0 2024-04-26T21:24:35.522 INFO:journalctl@ceph.mon.a.smithi046.stdout: index_shortening: 1 2024-04-26T21:24:35.522 INFO:journalctl@ceph.mon.a.smithi046.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-26T21:24:35.522 INFO:journalctl@ceph.mon.a.smithi046.stdout: checksum: 4 2024-04-26T21:24:35.522 INFO:journalctl@ceph.mon.a.smithi046.stdout: no_block_cache: 0 2024-04-26T21:24:35.522 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_cache: 0x5596088b71f0 2024-04-26T21:24:35.523 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_cache_name: BinnedLRUCache 2024-04-26T21:24:35.523 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_cache_options: 2024-04-26T21:24:35.523 INFO:journalctl@ceph.mon.a.smithi046.stdout: capacity : 536870912 2024-04-26T21:24:35.523 INFO:journalctl@ceph.mon.a.smithi046.stdout: num_shard_bits : 4 2024-04-26T21:24:35.523 INFO:journalctl@ceph.mon.a.smithi046.stdout: strict_capacity_limit : 0 2024-04-26T21:24:35.523 INFO:journalctl@ceph.mon.a.smithi046.stdout: high_pri_pool_ratio: 0.000 2024-04-26T21:24:35.523 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_cache_compressed: (nil) 2024-04-26T21:24:35.524 INFO:journalctl@ceph.mon.a.smithi046.stdout: persistent_cache: (nil) 2024-04-26T21:24:35.524 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_size: 4096 2024-04-26T21:24:35.524 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_size_deviation: 10 2024-04-26T21:24:35.524 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_restart_interval: 16 2024-04-26T21:24:35.524 INFO:journalctl@ceph.mon.a.smithi046.stdout: index_block_restart_interval: 1 2024-04-26T21:24:35.524 INFO:journalctl@ceph.mon.a.smithi046.stdout: metadata_block_size: 4096 2024-04-26T21:24:35.525 INFO:journalctl@ceph.mon.a.smithi046.stdout: partition_filters: 0 2024-04-26T21:24:35.525 INFO:journalctl@ceph.mon.a.smithi046.stdout: use_delta_encoding: 1 2024-04-26T21:24:35.525 INFO:journalctl@ceph.mon.a.smithi046.stdout: filter_policy: bloomfilter 2024-04-26T21:24:35.525 INFO:journalctl@ceph.mon.a.smithi046.stdout: whole_key_filtering: 1 2024-04-26T21:24:35.525 INFO:journalctl@ceph.mon.a.smithi046.stdout: verify_compression: 0 2024-04-26T21:24:35.525 INFO:journalctl@ceph.mon.a.smithi046.stdout: read_amp_bytes_per_bit: 0 2024-04-26T21:24:35.525 INFO:journalctl@ceph.mon.a.smithi046.stdout: format_version: 5 2024-04-26T21:24:35.526 INFO:journalctl@ceph.mon.a.smithi046.stdout: enable_index_compression: 1 2024-04-26T21:24:35.526 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_align: 0 2024-04-26T21:24:35.526 INFO:journalctl@ceph.mon.a.smithi046.stdout: max_auto_readahead_size: 262144 2024-04-26T21:24:35.526 INFO:journalctl@ceph.mon.a.smithi046.stdout: prepopulate_block_cache: 0 2024-04-26T21:24:35.526 INFO:journalctl@ceph.mon.a.smithi046.stdout: initial_auto_readahead_size: 8192 2024-04-26T21:24:35.526 INFO:journalctl@ceph.mon.a.smithi046.stdout: num_file_reads_for_auto_readahead: 2 2024-04-26T21:24:35.527 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.write_buffer_size: 33554432 2024-04-26T21:24:35.527 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.max_write_buffer_number: 2 2024-04-26T21:24:35.527 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.compression: NoCompression 2024-04-26T21:24:35.527 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.bottommost_compression: Disabled 2024-04-26T21:24:35.527 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.prefix_extractor: nullptr 2024-04-26T21:24:35.527 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-26T21:24:35.528 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.num_levels: 7 2024-04-26T21:24:35.528 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-26T21:24:35.528 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-26T21:24:35.528 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-26T21:24:35.528 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-26T21:24:35.528 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-26T21:24:35.529 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-26T21:24:35.529 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-26T21:24:35.529 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-26T21:24:35.529 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-26T21:24:35.529 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-26T21:24:35.529 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-26T21:24:35.530 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-26T21:24:35.530 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-26T21:24:35.530 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.compression_opts.level: 32767 2024-04-26T21:24:35.530 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.compression_opts.strategy: 0 2024-04-26T21:24:35.530 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-26T21:24:35.530 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-26T21:24:35.531 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-26T21:24:35.531 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-26T21:24:35.531 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.compression_opts.enabled: false 2024-04-26T21:24:35.531 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-26T21:24:35.531 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-26T21:24:35.532 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-26T21:24:35.532 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-26T21:24:35.532 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.target_file_size_base: 67108864 2024-04-26T21:24:35.532 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-26T21:24:35.532 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-26T21:24:35.532 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-26T21:24:35.532 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-26T21:24:35.533 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-26T21:24:35.533 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-26T21:24:35.535 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-26T21:24:35.535 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-26T21:24:35.535 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-26T21:24:35.535 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-26T21:24:35.535 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-26T21:24:35.535 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-26T21:24:35.536 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-26T21:24:35.536 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-26T21:24:35.536 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.arena_block_size: 1048576 2024-04-26T21:24:35.536 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-26T21:24:35.536 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-26T21:24:35.536 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.disable_auto_compactions: 0 2024-04-26T21:24:35.536 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-26T21:24:35.537 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-26T21:24:35.537 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-26T21:24:35.537 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-26T21:24:35.537 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-26T21:24:35.537 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-26T21:24:35.537 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-26T21:24:35.538 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-26T21:24:35.538 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-26T21:24:35.538 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-26T21:24:35.538 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-26T21:24:35.538 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.inplace_update_support: 0 2024-04-26T21:24:35.539 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-26T21:24:35.539 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-26T21:24:35.539 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-26T21:24:35.539 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-26T21:24:35.539 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.bloom_locality: 0 2024-04-26T21:24:35.539 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.max_successive_merges: 0 2024-04-26T21:24:35.539 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-26T21:24:35.540 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.paranoid_file_checks: 0 2024-04-26T21:24:35.540 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.force_consistency_checks: 1 2024-04-26T21:24:35.540 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.report_bg_io_stats: 0 2024-04-26T21:24:35.540 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.ttl: 2592000 2024-04-26T21:24:35.540 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-26T21:24:35.540 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-26T21:24:35.540 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-26T21:24:35.541 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.enable_blob_files: false 2024-04-26T21:24:35.541 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.min_blob_size: 0 2024-04-26T21:24:35.541 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.blob_file_size: 268435456 2024-04-26T21:24:35.541 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-26T21:24:35.541 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-26T21:24:35.542 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-26T21:24:35.542 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-26T21:24:35.542 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-26T21:24:35.542 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.blob_file_starting_level: 0 2024-04-26T21:24:35.542 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-26T21:24:35.542 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-26T21:24:35.543 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-26T21:24:35.543 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 60833427-0811-4183-8228-ea391f9d23ff 2024-04-26T21:24:35.543 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714166675272616, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-26T21:24:35.543 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-26T21:24:35.543 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714166675273800, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99850, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 324, "table_properties": {"data_size": 97736, "index_size": 306, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14058, "raw_average_key_size": 49, "raw_value_size": 90087, "raw_average_value_size": 318, "num_data_blocks": 12, "num_entries": 283, "num_filter_entries": 283, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714166675, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "60833427-0811-4183-8228-ea391f9d23ff", "db_session_id": "C2CG8KEX3FGPI3X23UNB", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-26T21:24:35.543 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714166675273889, "job": 1, "event": "recovery_finished"} 2024-04-26T21:24:35.544 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-26T21:24:35.544 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-26T21:24:35.544 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5596089d4000 2024-04-26T21:24:35.544 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: DB pointer 0x5596089c0000 2024-04-26T21:24:35.544 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-26T21:24:35.544 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-26T21:24:35.544 INFO:journalctl@ceph.mon.a.smithi046.stdout: ** DB Stats ** 2024-04-26T21:24:35.545 INFO:journalctl@ceph.mon.a.smithi046.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T21:24:35.545 INFO:journalctl@ceph.mon.a.smithi046.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-26T21:24:35.545 INFO:journalctl@ceph.mon.a.smithi046.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T21:24:35.545 INFO:journalctl@ceph.mon.a.smithi046.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T21:24:35.545 INFO:journalctl@ceph.mon.a.smithi046.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-26T21:24:35.545 INFO:journalctl@ceph.mon.a.smithi046.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T21:24:35.546 INFO:journalctl@ceph.mon.a.smithi046.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T21:24:35.546 INFO:journalctl@ceph.mon.a.smithi046.stdout: 2024-04-26T21:24:35.546 INFO:journalctl@ceph.mon.a.smithi046.stdout: ** Compaction Stats [default] ** 2024-04-26T21:24:35.546 INFO:journalctl@ceph.mon.a.smithi046.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T21:24:35.546 INFO:journalctl@ceph.mon.a.smithi046.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-26T21:24:35.547 INFO:journalctl@ceph.mon.a.smithi046.stdout: L0 2/0 99.37 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T21:24:35.547 INFO:journalctl@ceph.mon.a.smithi046.stdout: Sum 2/0 99.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T21:24:35.547 INFO:journalctl@ceph.mon.a.smithi046.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 127.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T21:24:35.547 INFO:journalctl@ceph.mon.a.smithi046.stdout: 2024-04-26T21:24:35.547 INFO:journalctl@ceph.mon.a.smithi046.stdout: ** Compaction Stats [default] ** 2024-04-26T21:24:35.547 INFO:journalctl@ceph.mon.a.smithi046.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T21:24:35.548 INFO:journalctl@ceph.mon.a.smithi046.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-26T21:24:35.548 INFO:journalctl@ceph.mon.a.smithi046.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 127.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T21:24:35.548 INFO:journalctl@ceph.mon.a.smithi046.stdout: 2024-04-26T21:24:35.548 INFO:journalctl@ceph.mon.a.smithi046.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-26T21:24:35.548 INFO:journalctl@ceph.mon.a.smithi046.stdout: 2024-04-26T21:24:35.548 INFO:journalctl@ceph.mon.a.smithi046.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T21:24:35.548 INFO:journalctl@ceph.mon.a.smithi046.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-26T21:24:35.549 INFO:journalctl@ceph.mon.a.smithi046.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-26T21:24:35.549 INFO:journalctl@ceph.mon.a.smithi046.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-26T21:24:35.549 INFO:journalctl@ceph.mon.a.smithi046.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-26T21:24:35.549 INFO:journalctl@ceph.mon.a.smithi046.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-26T21:24:35.549 INFO:journalctl@ceph.mon.a.smithi046.stdout: Cumulative compaction: 0.00 GB write, 29.34 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T21:24:35.549 INFO:journalctl@ceph.mon.a.smithi046.stdout: Interval compaction: 0.00 GB write, 29.34 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T21:24:35.550 INFO:journalctl@ceph.mon.a.smithi046.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-26T21:24:35.550 INFO:journalctl@ceph.mon.a.smithi046.stdout: Block cache BinnedLRUCache@0x5596088b71f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-26T21:24:35.550 INFO:journalctl@ceph.mon.a.smithi046.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-26T21:24:35.550 INFO:journalctl@ceph.mon.a.smithi046.stdout: 2024-04-26T21:24:35.550 INFO:journalctl@ceph.mon.a.smithi046.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-26T21:24:35.550 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: starting mon.a rank 0 at public addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] at bind addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:24:35.551 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: mon.a@-1(???) e1 preinit fsid 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:24:35.551 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: mon.a@-1(???).mds e1 new map 2024-04-26T21:24:35.551 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: mon.a@-1(???).mds e1 print_map 2024-04-26T21:24:35.551 INFO:journalctl@ceph.mon.a.smithi046.stdout: e1 2024-04-26T21:24:35.551 INFO:journalctl@ceph.mon.a.smithi046.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T21:24:35.551 INFO:journalctl@ceph.mon.a.smithi046.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T21:24:35.552 INFO:journalctl@ceph.mon.a.smithi046.stdout: legacy client fscid: -1 2024-04-26T21:24:35.552 INFO:journalctl@ceph.mon.a.smithi046.stdout: 2024-04-26T21:24:35.552 INFO:journalctl@ceph.mon.a.smithi046.stdout: No filesystems configured 2024-04-26T21:24:35.552 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T21:24:35.552 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T21:24:35.552 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T21:24:35.552 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T21:24:35.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-26T21:24:35.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 bash[29182]: fc596ea263bf65540b7a94fda1bb803cbcc1ec30cfa0880760818f6c09e7de32 2024-04-26T21:24:35.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 systemd[1]: Started Ceph mon.a for 36c7013a-0413-11ef-bc93-c7b262605968. 2024-04-26T21:24:35.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T21:24:35.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: monmap epoch 1 2024-04-26T21:24:35.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: fsid 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:24:35.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: last_changed 2024-04-26T21:24:27.425916+0000 2024-04-26T21:24:35.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: created 2024-04-26T21:24:27.425916+0000 2024-04-26T21:24:35.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: min_mon_release 19 (squid) 2024-04-26T21:24:35.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: election_strategy: 1 2024-04-26T21:24:35.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: 0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-04-26T21:24:35.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: fsmap 2024-04-26T21:24:35.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: osdmap e1: 0 total, 0 up, 0 in 2024-04-26T21:24:35.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:35 smithi046 ceph-mon[29201]: mgrmap e1: no daemons active 2024-04-26T21:24:36.621 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.776+0000 7f660fade700 1 Processor -- start 2024-04-26T21:24:36.621 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.776+0000 7f660fade700 1 -- start start 2024-04-26T21:24:36.621 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.776+0000 7f660fade700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6608108d90 0x7f66081091b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:36.621 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.776+0000 7f660fade700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6608109780 con 0x7f6608108d90 2024-04-26T21:24:36.622 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.776+0000 7f660d87a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6608108d90 0x7f66081091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:36.622 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.776+0000 7f660d87a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6608108d90 0x7f66081091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38924/0 (socket says 172.21.15.46:38924) 2024-04-26T21:24:36.622 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.776+0000 7f660d87a700 1 -- 172.21.15.46:0/4284845089 learned_addr learned my addr 172.21.15.46:0/4284845089 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:24:36.622 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.777+0000 7f660d87a700 1 -- 172.21.15.46:0/4284845089 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6608109f90 con 0x7f6608108d90 2024-04-26T21:24:36.622 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.777+0000 7f660d87a700 1 --2- 172.21.15.46:0/4284845089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6608108d90 0x7f66081091b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f66040098a0 tx=0x7f6604009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=719f374b51535b1e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:36.622 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.777+0000 7f660c878700 1 -- 172.21.15.46:0/4284845089 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6604004160 con 0x7f6608108d90 2024-04-26T21:24:36.622 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.777+0000 7f660c878700 1 -- 172.21.15.46:0/4284845089 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6604004300 con 0x7f6608108d90 2024-04-26T21:24:36.622 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.777+0000 7f660c878700 1 -- 172.21.15.46:0/4284845089 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6604011af0 con 0x7f6608108d90 2024-04-26T21:24:36.622 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.777+0000 7f660fade700 1 -- 172.21.15.46:0/4284845089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6608108d90 msgr2=0x7f66081091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:36.622 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.777+0000 7f660fade700 1 --2- 172.21.15.46:0/4284845089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6608108d90 0x7f66081091b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f66040098a0 tx=0x7f6604009bb0 comp rx=0 tx=0).stop 2024-04-26T21:24:36.622 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.778+0000 7f660fade700 1 -- 172.21.15.46:0/4284845089 shutdown_connections 2024-04-26T21:24:36.622 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.778+0000 7f660fade700 1 --2- 172.21.15.46:0/4284845089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6608108d90 0x7f66081091b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:36.622 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.778+0000 7f660fade700 1 -- 172.21.15.46:0/4284845089 >> 172.21.15.46:0/4284845089 conn(0x7f6608104310 msgr2=0x7f6608106770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:24:36.622 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.778+0000 7f660fade700 1 -- 172.21.15.46:0/4284845089 shutdown_connections 2024-04-26T21:24:36.623 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.778+0000 7f660fade700 1 -- 172.21.15.46:0/4284845089 wait complete. 2024-04-26T21:24:36.623 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.778+0000 7f660fade700 1 Processor -- start 2024-04-26T21:24:36.623 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.778+0000 7f660fade700 1 -- start start 2024-04-26T21:24:36.623 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.778+0000 7f660fade700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6608108d90 0x7f660819cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:36.623 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.778+0000 7f660fade700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f660810d340 con 0x7f6608108d90 2024-04-26T21:24:36.623 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.779+0000 7f660d87a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6608108d90 0x7f660819cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:36.623 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.779+0000 7f660d87a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6608108d90 0x7f660819cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38930/0 (socket says 172.21.15.46:38930) 2024-04-26T21:24:36.623 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.779+0000 7f660d87a700 1 -- 172.21.15.46:0/2066710264 learned_addr learned my addr 172.21.15.46:0/2066710264 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:24:36.623 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.779+0000 7f660d87a700 1 -- 172.21.15.46:0/2066710264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f660819d260 con 0x7f6608108d90 2024-04-26T21:24:36.623 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.779+0000 7f660d87a700 1 --2- 172.21.15.46:0/2066710264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6608108d90 0x7f660819cd20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f66040117b0 tx=0x7f6604011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:36.623 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.779+0000 7f65feffd700 1 -- 172.21.15.46:0/2066710264 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6604003af0 con 0x7f6608108d90 2024-04-26T21:24:36.623 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.779+0000 7f65feffd700 1 -- 172.21.15.46:0/2066710264 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6604020070 con 0x7f6608108d90 2024-04-26T21:24:36.623 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.779+0000 7f660fade700 1 -- 172.21.15.46:0/2066710264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f660819d4f0 con 0x7f6608108d90 2024-04-26T21:24:36.623 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.779+0000 7f65feffd700 1 -- 172.21.15.46:0/2066710264 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6604019630 con 0x7f6608108d90 2024-04-26T21:24:36.623 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.779+0000 7f660fade700 1 -- 172.21.15.46:0/2066710264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66081a01d0 con 0x7f6608108d90 2024-04-26T21:24:36.624 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.779+0000 7f65feffd700 1 -- 172.21.15.46:0/2066710264 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6604019c30 con 0x7f6608108d90 2024-04-26T21:24:36.624 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.780+0000 7f65feffd700 1 -- 172.21.15.46:0/2066710264 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f660400b720 con 0x7f6608108d90 2024-04-26T21:24:36.624 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.780+0000 7f660fade700 1 -- 172.21.15.46:0/2066710264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6608064a80 con 0x7f6608108d90 2024-04-26T21:24:36.624 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.783+0000 7f65feffd700 1 -- 172.21.15.46:0/2066710264 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f66040042a0 con 0x7f6608108d90 2024-04-26T21:24:36.624 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.854+0000 7f660fade700 1 -- 172.21.15.46:0/2066710264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f66081a03a0 con 0x7f6608108d90 2024-04-26T21:24:36.624 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.856+0000 7f65feffd700 1 -- 172.21.15.46:0/2066710264 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f660401a9c0 con 0x7f6608108d90 2024-04-26T21:24:36.624 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.856+0000 7f660fade700 1 -- 172.21.15.46:0/2066710264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6608108d90 msgr2=0x7f660819cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:36.624 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.856+0000 7f660fade700 1 --2- 172.21.15.46:0/2066710264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6608108d90 0x7f660819cd20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f66040117b0 tx=0x7f6604011b70 comp rx=0 tx=0).stop 2024-04-26T21:24:36.624 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.856+0000 7f660fade700 1 -- 172.21.15.46:0/2066710264 shutdown_connections 2024-04-26T21:24:36.624 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.856+0000 7f660fade700 1 --2- 172.21.15.46:0/2066710264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6608108d90 0x7f660819cd20 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:36.624 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.856+0000 7f660fade700 1 -- 172.21.15.46:0/2066710264 >> 172.21.15.46:0/2066710264 conn(0x7f6608104310 msgr2=0x7f6608104e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:24:36.624 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.856+0000 7f660fade700 1 -- 172.21.15.46:0/2066710264 shutdown_connections 2024-04-26T21:24:36.624 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:35.856+0000 7f660fade700 1 -- 172.21.15.46:0/2066710264 wait complete. 2024-04-26T21:24:36.624 INFO:teuthology.orchestra.run.smithi046.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-26T21:24:36.647 INFO:teuthology.orchestra.run.smithi046.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-26T21:24:36.647 INFO:teuthology.orchestra.run.smithi046.stdout:Creating mgr... 2024-04-26T21:24:36.647 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-26T21:24:36.648 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-26T21:24:36.904 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-36c7013a-0413-11ef-bc93-c7b262605968@mgr.x 2024-04-26T21:24:36.905 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Failed to reset failed state of unit ceph-36c7013a-0413-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-36c7013a-0413-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-26T21:24:37.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:36 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2066710264' entity='client.admin' 2024-04-26T21:24:37.057 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-36c7013a-0413-11ef-bc93-c7b262605968.target.wants/ceph-36c7013a-0413-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-36c7013a-0413-11ef-bc93-c7b262605968@.service. 2024-04-26T21:24:37.573 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T21:24:37.573 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout disabled 2024-04-26T21:24:37.580 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T21:24:37.580 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2024-04-26T21:24:37.581 INFO:teuthology.orchestra.run.smithi046.stdout:firewalld.service is not enabled 2024-04-26T21:24:37.581 INFO:teuthology.orchestra.run.smithi046.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T21:24:37.589 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T21:24:37.589 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout disabled 2024-04-26T21:24:37.597 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T21:24:37.597 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2024-04-26T21:24:37.598 INFO:teuthology.orchestra.run.smithi046.stdout:firewalld.service is not enabled 2024-04-26T21:24:37.598 INFO:teuthology.orchestra.run.smithi046.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-26T21:24:37.598 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mgr to start... 2024-04-26T21:24:37.598 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mgr... 2024-04-26T21:24:38.288 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:38 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/370970167' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T21:24:38.288 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:38 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:38.286+0000 7fd177786200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T21:24:38.553 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:38 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:38.375+0000 7fd177786200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T21:24:38.673 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-04-26T21:24:38.674 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-04-26T21:24:38.674 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsid": "36c7013a-0413-11ef-bc93-c7b262605968", 2024-04-26T21:24:38.674 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T21:24:38.674 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T21:24:38.674 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T21:24:38.674 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T21:24:38.674 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:24:38.674 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T21:24:38.674 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T21:24:38.674 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 0 2024-04-26T21:24:38.674 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-04-26T21:24:38.674 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T21:24:38.674 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T21:24:38.675 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-04-26T21:24:38.675 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-26T21:24:38.675 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T21:24:38.675 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:24:38.675 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T21:24:38.675 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T21:24:38.675 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:24:38.675 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T21:24:38.675 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:24:38.675 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T21:24:38.675 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T21:24:38.675 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T21:24:38.683 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T21:24:38.683 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T21:24:38.683 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T21:24:38.683 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:24:38.683 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T21:24:38.683 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T21:24:38.684 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T21:24:38.684 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T21:24:38.684 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T21:24:38.684 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T21:24:38.684 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T21:24:38.684 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T21:24:38.684 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T21:24:38.684 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:24:38.684 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T21:24:38.684 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:24:38.684 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T21:24:38.684 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T21:24:38.684 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:24:38.684 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T21:24:38.685 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T21:24:38.685 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T21:24:38.685 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T21:24:38.685 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T21:24:38.685 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T21:24:38.685 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T21:24:38.685 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-04-26T21:24:38.685 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T21:24:38.685 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:24:38.685 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T21:24:38.685 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:24:38.685 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T21:24:30.481655+0000", 2024-04-26T21:24:38.685 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T21:24:38.685 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:24:38.686 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T21:24:38.686 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-04-26T21:24:38.686 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.051+0000 7f92a63de700 1 Processor -- start 2024-04-26T21:24:38.686 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.051+0000 7f92a63de700 1 -- start start 2024-04-26T21:24:38.686 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.051+0000 7f92a63de700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92a0108dd0 0x7f92a01091f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:38.686 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.051+0000 7f92a63de700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f92a01097c0 con 0x7f92a0108dd0 2024-04-26T21:24:38.686 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.051+0000 7f929ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92a0108dd0 0x7f92a01091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:38.686 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.051+0000 7f929ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92a0108dd0 0x7f92a01091f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38968/0 (socket says 172.21.15.46:38968) 2024-04-26T21:24:38.686 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.051+0000 7f929ffff700 1 -- 172.21.15.46:0/1059067215 learned_addr learned my addr 172.21.15.46:0/1059067215 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:24:38.686 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.052+0000 7f929ffff700 1 -- 172.21.15.46:0/1059067215 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92a0109f30 con 0x7f92a0108dd0 2024-04-26T21:24:38.686 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.052+0000 7f929ffff700 1 --2- 172.21.15.46:0/1059067215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92a0108dd0 0x7f92a01091f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f92900098a0 tx=0x7f9290009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3f3a52391f19c20 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:38.686 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.052+0000 7f929effd700 1 -- 172.21.15.46:0/1059067215 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9290004160 con 0x7f92a0108dd0 2024-04-26T21:24:38.686 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.052+0000 7f929effd700 1 -- 172.21.15.46:0/1059067215 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9290004300 con 0x7f92a0108dd0 2024-04-26T21:24:38.687 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.052+0000 7f929effd700 1 -- 172.21.15.46:0/1059067215 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9290011af0 con 0x7f92a0108dd0 2024-04-26T21:24:38.687 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.053+0000 7f92a63de700 1 -- 172.21.15.46:0/1059067215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92a0108dd0 msgr2=0x7f92a01091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:38.687 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.053+0000 7f92a63de700 1 --2- 172.21.15.46:0/1059067215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92a0108dd0 0x7f92a01091f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f92900098a0 tx=0x7f9290009bb0 comp rx=0 tx=0).stop 2024-04-26T21:24:38.687 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.053+0000 7f92a63de700 1 -- 172.21.15.46:0/1059067215 shutdown_connections 2024-04-26T21:24:38.687 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.053+0000 7f92a63de700 1 --2- 172.21.15.46:0/1059067215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92a0108dd0 0x7f92a01091f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:38.687 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.053+0000 7f92a63de700 1 -- 172.21.15.46:0/1059067215 >> 172.21.15.46:0/1059067215 conn(0x7f92a0104370 msgr2=0x7f92a01067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:24:38.687 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.053+0000 7f92a63de700 1 -- 172.21.15.46:0/1059067215 shutdown_connections 2024-04-26T21:24:38.687 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.053+0000 7f92a63de700 1 -- 172.21.15.46:0/1059067215 wait complete. 2024-04-26T21:24:38.687 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.054+0000 7f92a63de700 1 Processor -- start 2024-04-26T21:24:38.687 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.054+0000 7f92a63de700 1 -- start start 2024-04-26T21:24:38.687 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.054+0000 7f92a63de700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92a0108dd0 0x7f92a019cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:38.687 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.054+0000 7f92a63de700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f92a010d2e0 con 0x7f92a0108dd0 2024-04-26T21:24:38.687 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.054+0000 7f929ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92a0108dd0 0x7f92a019cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:38.687 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.054+0000 7f929ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92a0108dd0 0x7f92a019cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38984/0 (socket says 172.21.15.46:38984) 2024-04-26T21:24:38.688 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.054+0000 7f929ffff700 1 -- 172.21.15.46:0/370970167 learned_addr learned my addr 172.21.15.46:0/370970167 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:24:38.688 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.055+0000 7f929ffff700 1 -- 172.21.15.46:0/370970167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92a019d250 con 0x7f92a0108dd0 2024-04-26T21:24:38.688 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.055+0000 7f929ffff700 1 --2- 172.21.15.46:0/370970167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92a0108dd0 0x7f92a019cd10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9290009fb0 tx=0x7f9290011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:38.688 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.055+0000 7f929d7fa700 1 -- 172.21.15.46:0/370970167 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92900036a0 con 0x7f92a0108dd0 2024-04-26T21:24:38.688 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.055+0000 7f929d7fa700 1 -- 172.21.15.46:0/370970167 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f929001a430 con 0x7f92a0108dd0 2024-04-26T21:24:38.688 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.055+0000 7f929d7fa700 1 -- 172.21.15.46:0/370970167 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9290019550 con 0x7f92a0108dd0 2024-04-26T21:24:38.688 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.055+0000 7f92a63de700 1 -- 172.21.15.46:0/370970167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92a019d4e0 con 0x7f92a0108dd0 2024-04-26T21:24:38.688 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.055+0000 7f92a63de700 1 -- 172.21.15.46:0/370970167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92a01a01c0 con 0x7f92a0108dd0 2024-04-26T21:24:38.688 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.055+0000 7f929d7fa700 1 -- 172.21.15.46:0/370970167 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9290019b90 con 0x7f92a0108dd0 2024-04-26T21:24:38.688 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.055+0000 7f929d7fa700 1 -- 172.21.15.46:0/370970167 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9290024730 con 0x7f92a0108dd0 2024-04-26T21:24:38.688 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.056+0000 7f92a63de700 1 -- 172.21.15.46:0/370970167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92a0064a80 con 0x7f92a0108dd0 2024-04-26T21:24:38.688 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.058+0000 7f929d7fa700 1 -- 172.21.15.46:0/370970167 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f929001a5e0 con 0x7f92a0108dd0 2024-04-26T21:24:38.688 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.133+0000 7f92a63de700 1 -- 172.21.15.46:0/370970167 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f92a01a0390 con 0x7f92a0108dd0 2024-04-26T21:24:38.688 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.133+0000 7f929d7fa700 1 -- 172.21.15.46:0/370970167 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f929001a8e0 con 0x7f92a0108dd0 2024-04-26T21:24:38.689 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.134+0000 7f92a63de700 1 -- 172.21.15.46:0/370970167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92a0108dd0 msgr2=0x7f92a019cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:38.689 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.134+0000 7f92a63de700 1 --2- 172.21.15.46:0/370970167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92a0108dd0 0x7f92a019cd10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9290009fb0 tx=0x7f9290011940 comp rx=0 tx=0).stop 2024-04-26T21:24:38.689 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.134+0000 7f92a63de700 1 -- 172.21.15.46:0/370970167 shutdown_connections 2024-04-26T21:24:38.689 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.134+0000 7f92a63de700 1 --2- 172.21.15.46:0/370970167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92a0108dd0 0x7f92a019cd10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:38.689 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.134+0000 7f92a63de700 1 -- 172.21.15.46:0/370970167 >> 172.21.15.46:0/370970167 conn(0x7f92a0104370 msgr2=0x7f92a0104f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:24:38.689 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.134+0000 7f92a63de700 1 -- 172.21.15.46:0/370970167 shutdown_connections 2024-04-26T21:24:38.689 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:38.134+0000 7f92a63de700 1 -- 172.21.15.46:0/370970167 wait complete. 2024-04-26T21:24:38.689 INFO:teuthology.orchestra.run.smithi046.stdout:mgr not available, waiting (1/15)... 2024-04-26T21:24:39.936 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:39 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:39.653+0000 7fd177786200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T21:24:39.937 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:39 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:39.831+0000 7fd177786200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T21:24:40.305 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:40 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:40.017+0000 7fd177786200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T21:24:40.626 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:40 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:40.315+0000 7fd177786200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T21:24:40.626 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:40 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:40.411+0000 7fd177786200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T21:24:40.627 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:40 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:40.625+0000 7fd177786200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T21:24:41.053 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:40 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:40.831+0000 7fd177786200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T21:24:41.803 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:41 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:41.468+0000 7fd177786200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T21:24:42.235 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:41 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:41.918+0000 7fd177786200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T21:24:42.235 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:42 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:42.022+0000 7fd177786200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T21:24:42.235 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:42 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:42.234+0000 7fd177786200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T21:24:42.962 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:42 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2087473717' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T21:24:42.963 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:42 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:42.805+0000 7fd177786200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T21:24:43.303 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:42 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:42.921+0000 7fd177786200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T21:24:43.303 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:43 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:43.177+0000 7fd177786200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T21:24:43.803 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:43 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:43.317+0000 7fd177786200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T21:24:43.803 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:43 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:43.418+0000 7fd177786200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T21:24:43.804 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:43 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:43.538+0000 7fd177786200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T21:24:44.303 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:43 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:43.851+0000 7fd177786200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T21:24:44.771 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:44 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:44.490+0000 7fd177786200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T21:24:44.771 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:44 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:44.717+0000 7fd177786200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T21:24:45.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:44 smithi046 ceph-mon[29201]: Activating manager daemon x 2024-04-26T21:24:45.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:44 smithi046 ceph-mon[29201]: mgrmap e2: x(active, starting, since 0.0030699s) 2024-04-26T21:24:45.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:44 smithi046 ceph-mon[29201]: from='mgr.14100 172.21.15.46:0/2228244316' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T21:24:45.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:44 smithi046 ceph-mon[29201]: from='mgr.14100 172.21.15.46:0/2228244316' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T21:24:45.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:44 smithi046 ceph-mon[29201]: from='mgr.14100 172.21.15.46:0/2228244316' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T21:24:45.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:44 smithi046 ceph-mon[29201]: from='mgr.14100 172.21.15.46:0/2228244316' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:24:45.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:44 smithi046 ceph-mon[29201]: from='mgr.14100 172.21.15.46:0/2228244316' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T21:24:45.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:44 smithi046 ceph-mon[29201]: Manager daemon x is now available 2024-04-26T21:24:45.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:44 smithi046 ceph-mon[29201]: from='mgr.14100 172.21.15.46:0/2228244316' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T21:24:45.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:44 smithi046 ceph-mon[29201]: from='mgr.14100 172.21.15.46:0/2228244316' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T21:24:45.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:44 smithi046 ceph-mon[29201]: from='mgr.14100 172.21.15.46:0/2228244316' entity='mgr.x' 2024-04-26T21:24:45.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:44 smithi046 ceph-mon[29201]: from='mgr.14100 172.21.15.46:0/2228244316' entity='mgr.x' 2024-04-26T21:24:45.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:44 smithi046 ceph-mon[29201]: from='mgr.14100 172.21.15.46:0/2228244316' entity='mgr.x' 2024-04-26T21:24:47.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:46 smithi046 ceph-mon[29201]: mgrmap e3: x(active, since 1.00756s) 2024-04-26T21:24:48.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:47 smithi046 ceph-mon[29201]: mgrmap e4: x(active, since 2s) 2024-04-26T21:24:49.238 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-04-26T21:24:49.238 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-04-26T21:24:49.238 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsid": "36c7013a-0413-11ef-bc93-c7b262605968", 2024-04-26T21:24:49.238 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T21:24:49.238 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T21:24:49.238 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T21:24:49.238 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T21:24:49.238 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:24:49.239 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T21:24:49.239 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T21:24:49.239 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 0 2024-04-26T21:24:49.239 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-04-26T21:24:49.239 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T21:24:49.239 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T21:24:49.239 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-04-26T21:24:49.239 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-04-26T21:24:49.239 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T21:24:49.239 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:24:49.239 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T21:24:49.239 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T21:24:49.239 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:24:49.239 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T21:24:49.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:24:49.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T21:24:49.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T21:24:49.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T21:24:49.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T21:24:49.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T21:24:49.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T21:24:49.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:24:49.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T21:24:49.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T21:24:49.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T21:24:49.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T21:24:49.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T21:24:49.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T21:24:49.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T21:24:49.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T21:24:49.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T21:24:49.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:24:49.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T21:24:49.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:24:49.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T21:24:49.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T21:24:49.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:24:49.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T21:24:49.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T21:24:49.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T21:24:49.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T21:24:49.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T21:24:49.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T21:24:49.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T21:24:49.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-04-26T21:24:49.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T21:24:49.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:24:49.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T21:24:49.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:24:49.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T21:24:30.481655+0000", 2024-04-26T21:24:49.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T21:24:49.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:24:49.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T21:24:49.242 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-04-26T21:24:49.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.811+0000 7fc95ccdd700 1 Processor -- start 2024-04-26T21:24:49.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.811+0000 7fc95ccdd700 1 -- start start 2024-04-26T21:24:49.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.812+0000 7fc95ccdd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc95807f6c0 0x7fc95807dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:49.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.812+0000 7fc95ccdd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc95807e100 con 0x7fc95807f6c0 2024-04-26T21:24:49.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.812+0000 7fc95659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc95807f6c0 0x7fc95807dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:49.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.812+0000 7fc95659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc95807f6c0 0x7fc95807dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38994/0 (socket says 172.21.15.46:38994) 2024-04-26T21:24:49.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.812+0000 7fc95659c700 1 -- 172.21.15.46:0/951920044 learned_addr learned my addr 172.21.15.46:0/951920044 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:24:49.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.812+0000 7fc95659c700 1 -- 172.21.15.46:0/951920044 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc95807e280 con 0x7fc95807f6c0 2024-04-26T21:24:49.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.812+0000 7fc95659c700 1 --2- 172.21.15.46:0/951920044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc95807f6c0 0x7fc95807dbc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc9480098a0 tx=0x7fc948009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce74b5e466d3e5d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:49.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.812+0000 7fc95559a700 1 -- 172.21.15.46:0/951920044 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc948004160 con 0x7fc95807f6c0 2024-04-26T21:24:49.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.812+0000 7fc95559a700 1 -- 172.21.15.46:0/951920044 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc948004300 con 0x7fc95807f6c0 2024-04-26T21:24:49.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.812+0000 7fc95559a700 1 -- 172.21.15.46:0/951920044 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc948011af0 con 0x7fc95807f6c0 2024-04-26T21:24:49.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.813+0000 7fc95ccdd700 1 -- 172.21.15.46:0/951920044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc95807f6c0 msgr2=0x7fc95807dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:49.243 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.813+0000 7fc95ccdd700 1 --2- 172.21.15.46:0/951920044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc95807f6c0 0x7fc95807dbc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc9480098a0 tx=0x7fc948009bb0 comp rx=0 tx=0).stop 2024-04-26T21:24:49.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.813+0000 7fc95ccdd700 1 -- 172.21.15.46:0/951920044 shutdown_connections 2024-04-26T21:24:49.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.813+0000 7fc95ccdd700 1 --2- 172.21.15.46:0/951920044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc95807f6c0 0x7fc95807dbc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:49.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.813+0000 7fc95ccdd700 1 -- 172.21.15.46:0/951920044 >> 172.21.15.46:0/951920044 conn(0x7fc958104370 msgr2=0x7fc9581067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:24:49.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.813+0000 7fc95ccdd700 1 -- 172.21.15.46:0/951920044 shutdown_connections 2024-04-26T21:24:49.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.813+0000 7fc95ccdd700 1 -- 172.21.15.46:0/951920044 wait complete. 2024-04-26T21:24:49.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.814+0000 7fc95ccdd700 1 Processor -- start 2024-04-26T21:24:49.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.814+0000 7fc95ccdd700 1 -- start start 2024-04-26T21:24:49.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.814+0000 7fc95ccdd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc95807f6c0 0x7fc95819eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:49.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.814+0000 7fc95ccdd700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc95810f2e0 con 0x7fc95807f6c0 2024-04-26T21:24:49.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.814+0000 7fc95659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc95807f6c0 0x7fc95819eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:49.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.814+0000 7fc95659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc95807f6c0 0x7fc95819eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39000/0 (socket says 172.21.15.46:39000) 2024-04-26T21:24:49.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.814+0000 7fc95659c700 1 -- 172.21.15.46:0/2087473717 learned_addr learned my addr 172.21.15.46:0/2087473717 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:24:49.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.814+0000 7fc95659c700 1 -- 172.21.15.46:0/2087473717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc95819f400 con 0x7fc95807f6c0 2024-04-26T21:24:49.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.815+0000 7fc95659c700 1 --2- 172.21.15.46:0/2087473717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc95807f6c0 0x7fc95819eec0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc948011ac0 tx=0x7fc948011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:49.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.815+0000 7fc9477fe700 1 -- 172.21.15.46:0/2087473717 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc948003970 con 0x7fc95807f6c0 2024-04-26T21:24:49.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.815+0000 7fc9477fe700 1 -- 172.21.15.46:0/2087473717 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc948003b10 con 0x7fc95807f6c0 2024-04-26T21:24:49.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.815+0000 7fc95ccdd700 1 -- 172.21.15.46:0/2087473717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc95819f690 con 0x7fc95807f6c0 2024-04-26T21:24:49.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.815+0000 7fc95ccdd700 1 -- 172.21.15.46:0/2087473717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9581a2380 con 0x7fc95807f6c0 2024-04-26T21:24:49.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.815+0000 7fc9477fe700 1 -- 172.21.15.46:0/2087473717 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc948024610 con 0x7fc95807f6c0 2024-04-26T21:24:49.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.815+0000 7fc9477fe700 1 -- 172.21.15.46:0/2087473717 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc948024bd0 con 0x7fc95807f6c0 2024-04-26T21:24:49.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.815+0000 7fc9477fe700 1 -- 172.21.15.46:0/2087473717 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc948019700 con 0x7fc95807f6c0 2024-04-26T21:24:49.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.816+0000 7fc95ccdd700 1 -- 172.21.15.46:0/2087473717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc958064a80 con 0x7fc95807f6c0 2024-04-26T21:24:49.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.818+0000 7fc9477fe700 1 -- 172.21.15.46:0/2087473717 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fc94801a9e0 con 0x7fc95807f6c0 2024-04-26T21:24:49.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.905+0000 7fc95ccdd700 1 -- 172.21.15.46:0/2087473717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fc9581a2550 con 0x7fc95807f6c0 2024-04-26T21:24:49.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.906+0000 7fc9477fe700 1 -- 172.21.15.46:0/2087473717 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fc94801a430 con 0x7fc95807f6c0 2024-04-26T21:24:49.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.907+0000 7fc95ccdd700 1 -- 172.21.15.46:0/2087473717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc95807f6c0 msgr2=0x7fc95819eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:49.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.907+0000 7fc95ccdd700 1 --2- 172.21.15.46:0/2087473717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc95807f6c0 0x7fc95819eec0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc948011ac0 tx=0x7fc948011ba0 comp rx=0 tx=0).stop 2024-04-26T21:24:49.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.907+0000 7fc95ccdd700 1 -- 172.21.15.46:0/2087473717 shutdown_connections 2024-04-26T21:24:49.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.907+0000 7fc95ccdd700 1 --2- 172.21.15.46:0/2087473717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc95807f6c0 0x7fc95819eec0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:49.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.907+0000 7fc95ccdd700 1 -- 172.21.15.46:0/2087473717 >> 172.21.15.46:0/2087473717 conn(0x7fc958104370 msgr2=0x7fc958104f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:24:49.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.907+0000 7fc95ccdd700 1 -- 172.21.15.46:0/2087473717 shutdown_connections 2024-04-26T21:24:49.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:42.907+0000 7fc95ccdd700 1 -- 172.21.15.46:0/2087473717 wait complete. 2024-04-26T21:24:49.246 INFO:teuthology.orchestra.run.smithi046.stdout:mgr not available, waiting (2/15)... 2024-04-26T21:24:52.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:52 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/514138314' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T21:24:53.017 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-04-26T21:24:53.017 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-04-26T21:24:53.017 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsid": "36c7013a-0413-11ef-bc93-c7b262605968", 2024-04-26T21:24:53.018 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T21:24:53.018 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T21:24:53.018 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T21:24:53.018 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T21:24:53.018 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:24:53.018 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T21:24:53.018 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T21:24:53.018 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 0 2024-04-26T21:24:53.018 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-04-26T21:24:53.018 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T21:24:53.018 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T21:24:53.018 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-04-26T21:24:53.018 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_age": 17, 2024-04-26T21:24:53.018 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T21:24:53.019 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:24:53.019 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T21:24:53.019 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T21:24:53.019 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:24:53.019 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T21:24:53.019 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:24:53.019 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T21:24:53.019 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T21:24:53.027 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T21:24:53.028 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T21:24:53.028 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T21:24:53.028 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T21:24:53.028 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:24:53.028 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T21:24:53.028 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T21:24:53.028 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T21:24:53.028 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T21:24:53.028 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T21:24:53.028 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T21:24:53.028 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T21:24:53.028 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T21:24:53.028 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T21:24:53.029 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:24:53.029 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T21:24:53.029 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:24:53.029 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T21:24:53.029 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T21:24:53.029 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:24:53.029 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T21:24:53.029 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T21:24:53.029 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T21:24:53.029 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T21:24:53.029 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T21:24:53.029 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T21:24:53.029 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T21:24:53.029 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-04-26T21:24:53.029 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T21:24:53.030 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:24:53.030 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T21:24:53.030 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:24:53.030 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T21:24:30.481655+0000", 2024-04-26T21:24:53.030 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T21:24:53.030 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:24:53.030 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T21:24:53.030 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-04-26T21:24:53.030 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.109+0000 7fe9370d6700 1 Processor -- start 2024-04-26T21:24:53.030 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.109+0000 7fe9370d6700 1 -- start start 2024-04-26T21:24:53.030 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.109+0000 7fe9370d6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe930109800 0x7fe93010bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:53.030 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.109+0000 7fe9370d6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe93007d6e0 con 0x7fe930109800 2024-04-26T21:24:53.030 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.109+0000 7fe934e72700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe930109800 0x7fe93010bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:53.030 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.110+0000 7fe934e72700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe930109800 0x7fe93010bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54702/0 (socket says 172.21.15.46:54702) 2024-04-26T21:24:53.031 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.110+0000 7fe934e72700 1 -- 172.21.15.46:0/889707923 learned_addr learned my addr 172.21.15.46:0/889707923 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:24:53.031 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.110+0000 7fe934e72700 1 -- 172.21.15.46:0/889707923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe93010c130 con 0x7fe930109800 2024-04-26T21:24:53.031 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.110+0000 7fe934e72700 1 --2- 172.21.15.46:0/889707923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe930109800 0x7fe93010bbf0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe9240098a0 tx=0x7fe924009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c37965caa0316508 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:53.031 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.110+0000 7fe92f7fe700 1 -- 172.21.15.46:0/889707923 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe924004160 con 0x7fe930109800 2024-04-26T21:24:53.031 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.110+0000 7fe92f7fe700 1 -- 172.21.15.46:0/889707923 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe924004300 con 0x7fe930109800 2024-04-26T21:24:53.031 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.110+0000 7fe92f7fe700 1 -- 172.21.15.46:0/889707923 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe924011af0 con 0x7fe930109800 2024-04-26T21:24:53.031 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.111+0000 7fe9370d6700 1 -- 172.21.15.46:0/889707923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe930109800 msgr2=0x7fe93010bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:53.031 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.111+0000 7fe9370d6700 1 --2- 172.21.15.46:0/889707923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe930109800 0x7fe93010bbf0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe9240098a0 tx=0x7fe924009bb0 comp rx=0 tx=0).stop 2024-04-26T21:24:53.031 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.111+0000 7fe9370d6700 1 -- 172.21.15.46:0/889707923 shutdown_connections 2024-04-26T21:24:53.031 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.111+0000 7fe9370d6700 1 --2- 172.21.15.46:0/889707923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe930109800 0x7fe93010bbf0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:53.031 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.111+0000 7fe9370d6700 1 -- 172.21.15.46:0/889707923 >> 172.21.15.46:0/889707923 conn(0x7fe930103230 msgr2=0x7fe930105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:24:53.031 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.111+0000 7fe9370d6700 1 -- 172.21.15.46:0/889707923 shutdown_connections 2024-04-26T21:24:53.031 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.111+0000 7fe9370d6700 1 -- 172.21.15.46:0/889707923 wait complete. 2024-04-26T21:24:53.032 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.112+0000 7fe9370d6700 1 Processor -- start 2024-04-26T21:24:53.032 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.112+0000 7fe9370d6700 1 -- start start 2024-04-26T21:24:53.032 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.112+0000 7fe9370d6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe930109800 0x7fe93019cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:53.032 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.112+0000 7fe9370d6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe93010d0f0 con 0x7fe930109800 2024-04-26T21:24:53.032 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.112+0000 7fe934e72700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe930109800 0x7fe93019cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:53.032 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.112+0000 7fe934e72700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe930109800 0x7fe93019cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54706/0 (socket says 172.21.15.46:54706) 2024-04-26T21:24:53.032 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.112+0000 7fe934e72700 1 -- 172.21.15.46:0/514138314 learned_addr learned my addr 172.21.15.46:0/514138314 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:24:53.032 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.112+0000 7fe934e72700 1 -- 172.21.15.46:0/514138314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe93019d1d0 con 0x7fe930109800 2024-04-26T21:24:53.032 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.113+0000 7fe934e72700 1 --2- 172.21.15.46:0/514138314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe930109800 0x7fe93019cc90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe924011ac0 tx=0x7fe924011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:53.032 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.113+0000 7fe92dffb700 1 -- 172.21.15.46:0/514138314 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe924003970 con 0x7fe930109800 2024-04-26T21:24:53.032 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.113+0000 7fe92dffb700 1 -- 172.21.15.46:0/514138314 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe924003b10 con 0x7fe930109800 2024-04-26T21:24:53.032 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.113+0000 7fe9370d6700 1 -- 172.21.15.46:0/514138314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe93019d460 con 0x7fe930109800 2024-04-26T21:24:53.032 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.113+0000 7fe92dffb700 1 -- 172.21.15.46:0/514138314 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe924024610 con 0x7fe930109800 2024-04-26T21:24:53.033 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.113+0000 7fe9370d6700 1 -- 172.21.15.46:0/514138314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9301a0150 con 0x7fe930109800 2024-04-26T21:24:53.033 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.114+0000 7fe92dffb700 1 -- 172.21.15.46:0/514138314 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fe924024890 con 0x7fe930109800 2024-04-26T21:24:53.033 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.114+0000 7fe9370d6700 1 -- 172.21.15.46:0/514138314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe930064a80 con 0x7fe930109800 2024-04-26T21:24:53.033 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.114+0000 7fe92dffb700 1 --2- 172.21.15.46:0/514138314 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7fe920039560 0x7fe92003ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:53.033 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.114+0000 7fe92dffb700 1 -- 172.21.15.46:0/514138314 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe92404e840 con 0x7fe930109800 2024-04-26T21:24:53.033 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.114+0000 7fe92ffff700 1 --2- 172.21.15.46:0/514138314 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7fe920039560 0x7fe92003ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:53.033 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.115+0000 7fe92ffff700 1 --2- 172.21.15.46:0/514138314 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7fe920039560 0x7fe92003ba20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe91c006fd0 tx=0x7fe91c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:53.033 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.120+0000 7fe92dffb700 1 -- 172.21.15.46:0/514138314 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe9240172b0 con 0x7fe930109800 2024-04-26T21:24:53.033 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.418+0000 7fe9370d6700 1 -- 172.21.15.46:0/514138314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fe9301a0370 con 0x7fe930109800 2024-04-26T21:24:53.033 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.418+0000 7fe92dffb700 1 -- 172.21.15.46:0/514138314 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fe9301a0370 con 0x7fe930109800 2024-04-26T21:24:53.033 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.421+0000 7fe9370d6700 1 -- 172.21.15.46:0/514138314 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7fe920039560 msgr2=0x7fe92003ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:53.033 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.421+0000 7fe9370d6700 1 --2- 172.21.15.46:0/514138314 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7fe920039560 0x7fe92003ba20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe91c006fd0 tx=0x7fe91c006e40 comp rx=0 tx=0).stop 2024-04-26T21:24:53.033 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.421+0000 7fe9370d6700 1 -- 172.21.15.46:0/514138314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe930109800 msgr2=0x7fe93019cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:53.033 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.421+0000 7fe9370d6700 1 --2- 172.21.15.46:0/514138314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe930109800 0x7fe93019cc90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe924011ac0 tx=0x7fe924011ba0 comp rx=0 tx=0).stop 2024-04-26T21:24:53.034 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.421+0000 7fe9370d6700 1 -- 172.21.15.46:0/514138314 shutdown_connections 2024-04-26T21:24:53.034 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.421+0000 7fe9370d6700 1 --2- 172.21.15.46:0/514138314 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7fe920039560 0x7fe92003ba20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:53.034 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.421+0000 7fe9370d6700 1 --2- 172.21.15.46:0/514138314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe930109800 0x7fe93019cc90 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:53.034 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.421+0000 7fe9370d6700 1 -- 172.21.15.46:0/514138314 >> 172.21.15.46:0/514138314 conn(0x7fe930103230 msgr2=0x7fe930103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:24:53.034 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.421+0000 7fe9370d6700 1 -- 172.21.15.46:0/514138314 shutdown_connections 2024-04-26T21:24:53.034 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:52.421+0000 7fe9370d6700 1 -- 172.21.15.46:0/514138314 wait complete. 2024-04-26T21:24:53.034 INFO:teuthology.orchestra.run.smithi046.stdout:mgr is available 2024-04-26T21:24:53.983 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:53 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/239532576' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T21:24:54.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-04-26T21:24:54.244 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T21:24:54.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout fsid = 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:24:54.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-26T21:24:54.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T21:24:54.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T21:24:54.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T21:24:54.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T21:24:54.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-04-26T21:24:54.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T21:24:54.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T21:24:54.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-04-26T21:24:54.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T21:24:54.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-26T21:24:54.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-26T21:24:54.245 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-26T21:24:54.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T21:24:54.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T21:24:54.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.465+0000 7f6638ad6700 1 Processor -- start 2024-04-26T21:24:54.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.465+0000 7f6638ad6700 1 -- start start 2024-04-26T21:24:54.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.466+0000 7f6638ad6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6634109820 0x7f663410bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:54.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.466+0000 7f6638ad6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f663407ce80 con 0x7f6634109820 2024-04-26T21:24:54.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.466+0000 7f663259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6634109820 0x7f663410bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:54.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.466+0000 7f663259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6634109820 0x7f663410bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54716/0 (socket says 172.21.15.46:54716) 2024-04-26T21:24:54.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.466+0000 7f663259c700 1 -- 172.21.15.46:0/691769991 learned_addr learned my addr 172.21.15.46:0/691769991 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:24:54.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.466+0000 7f663259c700 1 -- 172.21.15.46:0/691769991 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f663410c150 con 0x7f6634109820 2024-04-26T21:24:54.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.467+0000 7f663259c700 1 --2- 172.21.15.46:0/691769991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6634109820 0x7f663410bc10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f66240098a0 tx=0x7f6624009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eac1a754f4b1dc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:54.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.467+0000 7f663159a700 1 -- 172.21.15.46:0/691769991 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6624004160 con 0x7f6634109820 2024-04-26T21:24:54.246 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.467+0000 7f663159a700 1 -- 172.21.15.46:0/691769991 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6624004300 con 0x7f6634109820 2024-04-26T21:24:54.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.467+0000 7f663159a700 1 -- 172.21.15.46:0/691769991 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6624011af0 con 0x7f6634109820 2024-04-26T21:24:54.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.467+0000 7f6638ad6700 1 -- 172.21.15.46:0/691769991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6634109820 msgr2=0x7f663410bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:54.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.467+0000 7f6638ad6700 1 --2- 172.21.15.46:0/691769991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6634109820 0x7f663410bc10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f66240098a0 tx=0x7f6624009bb0 comp rx=0 tx=0).stop 2024-04-26T21:24:54.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.467+0000 7f6638ad6700 1 -- 172.21.15.46:0/691769991 shutdown_connections 2024-04-26T21:24:54.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.467+0000 7f6638ad6700 1 --2- 172.21.15.46:0/691769991 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6634109820 0x7f663410bc10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:54.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.468+0000 7f6638ad6700 1 -- 172.21.15.46:0/691769991 >> 172.21.15.46:0/691769991 conn(0x7f66341031f0 msgr2=0x7f6634105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:24:54.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.468+0000 7f6638ad6700 1 -- 172.21.15.46:0/691769991 shutdown_connections 2024-04-26T21:24:54.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.468+0000 7f6638ad6700 1 -- 172.21.15.46:0/691769991 wait complete. 2024-04-26T21:24:54.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.468+0000 7f6638ad6700 1 Processor -- start 2024-04-26T21:24:54.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.468+0000 7f6638ad6700 1 -- start start 2024-04-26T21:24:54.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.469+0000 7f6638ad6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6634109820 0x7f663419cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:54.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.469+0000 7f6638ad6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f663410d0a0 con 0x7f6634109820 2024-04-26T21:24:54.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.469+0000 7f663259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6634109820 0x7f663419cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:54.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.469+0000 7f663259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6634109820 0x7f663419cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54720/0 (socket says 172.21.15.46:54720) 2024-04-26T21:24:54.247 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.469+0000 7f663259c700 1 -- 172.21.15.46:0/239532576 learned_addr learned my addr 172.21.15.46:0/239532576 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:24:54.248 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.469+0000 7f663259c700 1 -- 172.21.15.46:0/239532576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f663419d1c0 con 0x7f6634109820 2024-04-26T21:24:54.248 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.469+0000 7f663259c700 1 --2- 172.21.15.46:0/239532576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6634109820 0x7f663419cc80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6624011ac0 tx=0x7f6624011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:54.248 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.469+0000 7f662b7fe700 1 -- 172.21.15.46:0/239532576 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6624003970 con 0x7f6634109820 2024-04-26T21:24:54.248 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.469+0000 7f662b7fe700 1 -- 172.21.15.46:0/239532576 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6624003b10 con 0x7f6634109820 2024-04-26T21:24:54.248 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.469+0000 7f6638ad6700 1 -- 172.21.15.46:0/239532576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f663419d4b0 con 0x7f6634109820 2024-04-26T21:24:54.248 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.469+0000 7f662b7fe700 1 -- 172.21.15.46:0/239532576 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6624024610 con 0x7f6634109820 2024-04-26T21:24:54.248 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.469+0000 7f6638ad6700 1 -- 172.21.15.46:0/239532576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66341a0140 con 0x7f6634109820 2024-04-26T21:24:54.248 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.470+0000 7f662b7fe700 1 -- 172.21.15.46:0/239532576 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f6624024840 con 0x7f6634109820 2024-04-26T21:24:54.248 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.470+0000 7f6638ad6700 1 -- 172.21.15.46:0/239532576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6634064a80 con 0x7f6634109820 2024-04-26T21:24:54.248 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.470+0000 7f662b7fe700 1 --2- 172.21.15.46:0/239532576 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7f66200395b0 0x7f662003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:54.248 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.470+0000 7f662b7fe700 1 -- 172.21.15.46:0/239532576 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f662404d990 con 0x7f6634109820 2024-04-26T21:24:54.248 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.471+0000 7f6631d9b700 1 --2- 172.21.15.46:0/239532576 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7f66200395b0 0x7f662003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:54.248 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.471+0000 7f6631d9b700 1 --2- 172.21.15.46:0/239532576 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7f66200395b0 0x7f662003ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f661c006fd0 tx=0x7f661c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:54.248 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.475+0000 7f662b7fe700 1 -- 172.21.15.46:0/239532576 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f662404d3b0 con 0x7f6634109820 2024-04-26T21:24:54.248 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.682+0000 7f6638ad6700 1 -- 172.21.15.46:0/239532576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f66341a0310 con 0x7f6634109820 2024-04-26T21:24:54.249 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.683+0000 7f662b7fe700 1 -- 172.21.15.46:0/239532576 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7f662402d770 con 0x7f6634109820 2024-04-26T21:24:54.249 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.685+0000 7f6638ad6700 1 -- 172.21.15.46:0/239532576 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7f66200395b0 msgr2=0x7f662003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:54.249 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.685+0000 7f6638ad6700 1 --2- 172.21.15.46:0/239532576 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7f66200395b0 0x7f662003ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f661c006fd0 tx=0x7f661c006e40 comp rx=0 tx=0).stop 2024-04-26T21:24:54.249 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.685+0000 7f6638ad6700 1 -- 172.21.15.46:0/239532576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6634109820 msgr2=0x7f663419cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:54.249 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.685+0000 7f6638ad6700 1 --2- 172.21.15.46:0/239532576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6634109820 0x7f663419cc80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6624011ac0 tx=0x7f6624011ba0 comp rx=0 tx=0).stop 2024-04-26T21:24:54.249 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.685+0000 7f6638ad6700 1 -- 172.21.15.46:0/239532576 shutdown_connections 2024-04-26T21:24:54.249 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.685+0000 7f6638ad6700 1 --2- 172.21.15.46:0/239532576 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7f66200395b0 0x7f662003ba70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:54.249 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.685+0000 7f6638ad6700 1 --2- 172.21.15.46:0/239532576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6634109820 0x7f663419cc80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:54.249 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.685+0000 7f6638ad6700 1 -- 172.21.15.46:0/239532576 >> 172.21.15.46:0/239532576 conn(0x7f66341031f0 msgr2=0x7f6634103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:24:54.249 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.685+0000 7f6638ad6700 1 -- 172.21.15.46:0/239532576 shutdown_connections 2024-04-26T21:24:54.249 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:53.686+0000 7f6638ad6700 1 -- 172.21.15.46:0/239532576 wait complete. 2024-04-26T21:24:54.249 INFO:teuthology.orchestra.run.smithi046.stdout:Enabling cephadm module... 2024-04-26T21:24:55.282 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:54 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3213283846' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-26T21:24:55.282 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:55 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: ignoring --setuser ceph since I am not root 2024-04-26T21:24:55.282 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:55 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: ignoring --setgroup ceph since I am not root 2024-04-26T21:24:55.282 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:55 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:55.153+0000 7f4be1e05200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T21:24:55.553 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:55 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:55.344+0000 7f4be1e05200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T21:24:55.554 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:55 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:55.450+0000 7f4be1e05200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T21:24:55.565 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.650+0000 7fa8722fa700 1 Processor -- start 2024-04-26T21:24:55.565 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.650+0000 7fa8722fa700 1 -- start start 2024-04-26T21:24:55.565 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.650+0000 7fa8722fa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa86c107080 0x7fa86c1074a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:55.565 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.651+0000 7fa8722fa700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa86c107a70 con 0x7fa86c107080 2024-04-26T21:24:55.565 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.651+0000 7fa86bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa86c107080 0x7fa86c1074a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:55.566 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.651+0000 7fa86bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa86c107080 0x7fa86c1074a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54732/0 (socket says 172.21.15.46:54732) 2024-04-26T21:24:55.566 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.651+0000 7fa86bfff700 1 -- 172.21.15.46:0/2411034288 learned_addr learned my addr 172.21.15.46:0/2411034288 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:24:55.566 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.651+0000 7fa86bfff700 1 -- 172.21.15.46:0/2411034288 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa86c10a010 con 0x7fa86c107080 2024-04-26T21:24:55.566 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.651+0000 7fa86bfff700 1 --2- 172.21.15.46:0/2411034288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa86c107080 0x7fa86c1074a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa85c0098a0 tx=0x7fa85c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c1bdfbde38d4568 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:55.566 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.651+0000 7fa86b7fe700 1 -- 172.21.15.46:0/2411034288 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa85c004160 con 0x7fa86c107080 2024-04-26T21:24:55.566 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.651+0000 7fa86b7fe700 1 -- 172.21.15.46:0/2411034288 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa85c004300 con 0x7fa86c107080 2024-04-26T21:24:55.566 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.651+0000 7fa86b7fe700 1 -- 172.21.15.46:0/2411034288 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa85c011af0 con 0x7fa86c107080 2024-04-26T21:24:55.566 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.652+0000 7fa8722fa700 1 -- 172.21.15.46:0/2411034288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa86c107080 msgr2=0x7fa86c1074a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:55.566 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.652+0000 7fa8722fa700 1 --2- 172.21.15.46:0/2411034288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa86c107080 0x7fa86c1074a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa85c0098a0 tx=0x7fa85c009bb0 comp rx=0 tx=0).stop 2024-04-26T21:24:55.566 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.652+0000 7fa8722fa700 1 -- 172.21.15.46:0/2411034288 shutdown_connections 2024-04-26T21:24:55.566 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.652+0000 7fa8722fa700 1 --2- 172.21.15.46:0/2411034288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa86c107080 0x7fa86c1074a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:55.566 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.652+0000 7fa8722fa700 1 -- 172.21.15.46:0/2411034288 >> 172.21.15.46:0/2411034288 conn(0x7fa86c102970 msgr2=0x7fa86c104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:24:55.567 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.652+0000 7fa8722fa700 1 -- 172.21.15.46:0/2411034288 shutdown_connections 2024-04-26T21:24:55.567 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.652+0000 7fa8722fa700 1 -- 172.21.15.46:0/2411034288 wait complete. 2024-04-26T21:24:55.567 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.653+0000 7fa8722fa700 1 Processor -- start 2024-04-26T21:24:55.567 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.653+0000 7fa8722fa700 1 -- start start 2024-04-26T21:24:55.567 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.653+0000 7fa8722fa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa86c107080 0x7fa86c109890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:55.567 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.653+0000 7fa8722fa700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa86c10d3c0 con 0x7fa86c107080 2024-04-26T21:24:55.567 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.653+0000 7fa86bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa86c107080 0x7fa86c109890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:55.567 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.653+0000 7fa86bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa86c107080 0x7fa86c109890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54734/0 (socket says 172.21.15.46:54734) 2024-04-26T21:24:55.567 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.653+0000 7fa86bfff700 1 -- 172.21.15.46:0/3213283846 learned_addr learned my addr 172.21.15.46:0/3213283846 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:24:55.567 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.653+0000 7fa86bfff700 1 -- 172.21.15.46:0/3213283846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa86c109dd0 con 0x7fa86c107080 2024-04-26T21:24:55.567 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.653+0000 7fa86bfff700 1 --2- 172.21.15.46:0/3213283846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa86c107080 0x7fa86c109890 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa85c0117b0 tx=0x7fa85c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:55.567 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.653+0000 7fa869ffb700 1 -- 172.21.15.46:0/3213283846 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa85c003af0 con 0x7fa86c107080 2024-04-26T21:24:55.567 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.653+0000 7fa869ffb700 1 -- 172.21.15.46:0/3213283846 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa85c020070 con 0x7fa86c107080 2024-04-26T21:24:55.567 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.653+0000 7fa869ffb700 1 -- 172.21.15.46:0/3213283846 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa85c019630 con 0x7fa86c107080 2024-04-26T21:24:55.567 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.653+0000 7fa8722fa700 1 -- 172.21.15.46:0/3213283846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa86c107fc0 con 0x7fa86c107080 2024-04-26T21:24:55.568 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.654+0000 7fa8722fa700 1 -- 172.21.15.46:0/3213283846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa86c108500 con 0x7fa86c107080 2024-04-26T21:24:55.568 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.654+0000 7fa869ffb700 1 -- 172.21.15.46:0/3213283846 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fa85c01a430 con 0x7fa86c107080 2024-04-26T21:24:55.568 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.654+0000 7fa8722fa700 1 -- 172.21.15.46:0/3213283846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa86c064a80 con 0x7fa86c107080 2024-04-26T21:24:55.568 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.655+0000 7fa869ffb700 1 --2- 172.21.15.46:0/3213283846 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7fa8580395b0 0x7fa85803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:55.568 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.655+0000 7fa869ffb700 1 -- 172.21.15.46:0/3213283846 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa85c04dae0 con 0x7fa86c107080 2024-04-26T21:24:55.568 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.655+0000 7fa863fff700 1 --2- 172.21.15.46:0/3213283846 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7fa8580395b0 0x7fa85803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:55.568 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.655+0000 7fa863fff700 1 --2- 172.21.15.46:0/3213283846 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7fa8580395b0 0x7fa85803ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa854006fd0 tx=0x7fa854006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:55.568 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.660+0000 7fa869ffb700 1 -- 172.21.15.46:0/3213283846 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa85c00e260 con 0x7fa86c107080 2024-04-26T21:24:55.568 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.929+0000 7fa8722fa700 1 -- 172.21.15.46:0/3213283846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fa86c108780 con 0x7fa86c107080 2024-04-26T21:24:55.568 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.980+0000 7fa869ffb700 1 -- 172.21.15.46:0/3213283846 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fa85c024e80 con 0x7fa86c107080 2024-04-26T21:24:55.568 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.982+0000 7fa869ffb700 1 -- 172.21.15.46:0/3213283846 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fa85c019850 con 0x7fa86c107080 2024-04-26T21:24:55.568 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.983+0000 7fa8722fa700 1 -- 172.21.15.46:0/3213283846 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7fa8580395b0 msgr2=0x7fa85803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:55.568 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.983+0000 7fa8722fa700 1 --2- 172.21.15.46:0/3213283846 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7fa8580395b0 0x7fa85803ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa854006fd0 tx=0x7fa854006e40 comp rx=0 tx=0).stop 2024-04-26T21:24:55.568 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.983+0000 7fa8722fa700 1 -- 172.21.15.46:0/3213283846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa86c107080 msgr2=0x7fa86c109890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:55.569 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.983+0000 7fa8722fa700 1 --2- 172.21.15.46:0/3213283846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa86c107080 0x7fa86c109890 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa85c0117b0 tx=0x7fa85c011b70 comp rx=0 tx=0).stop 2024-04-26T21:24:55.569 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.983+0000 7fa8722fa700 1 -- 172.21.15.46:0/3213283846 shutdown_connections 2024-04-26T21:24:55.569 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.983+0000 7fa8722fa700 1 --2- 172.21.15.46:0/3213283846 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7fa8580395b0 0x7fa85803ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:55.569 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.983+0000 7fa8722fa700 1 --2- 172.21.15.46:0/3213283846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa86c107080 0x7fa86c109890 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:55.569 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.983+0000 7fa8722fa700 1 -- 172.21.15.46:0/3213283846 >> 172.21.15.46:0/3213283846 conn(0x7fa86c102970 msgr2=0x7fa86c103470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:24:55.569 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.983+0000 7fa8722fa700 1 -- 172.21.15.46:0/3213283846 shutdown_connections 2024-04-26T21:24:55.569 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:54.983+0000 7fa8722fa700 1 -- 172.21.15.46:0/3213283846 wait complete. 2024-04-26T21:24:55.824 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:55 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:55.774+0000 7f4be1e05200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T21:24:56.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:55 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3213283846' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-26T21:24:56.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:55 smithi046 ceph-mon[29201]: mgrmap e5: x(active, since 10s) 2024-04-26T21:24:56.226 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:55 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:55.865+0000 7f4be1e05200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T21:24:56.863 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-04-26T21:24:56.863 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-26T21:24:56.863 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T21:24:56.863 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-26T21:24:56.864 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T21:24:56.864 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-04-26T21:24:56.864 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.020+0000 7f374fe03700 1 Processor -- start 2024-04-26T21:24:56.864 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.020+0000 7f374fe03700 1 -- start start 2024-04-26T21:24:56.864 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.020+0000 7f374fe03700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3748109860 0x7f374810bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:56.864 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.020+0000 7f374fe03700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f374807ce40 con 0x7f3748109860 2024-04-26T21:24:56.864 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.020+0000 7f374db9f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3748109860 0x7f374810bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:56.864 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.020+0000 7f374db9f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3748109860 0x7f374810bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42814/0 (socket says 172.21.15.46:42814) 2024-04-26T21:24:56.864 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.020+0000 7f374db9f700 1 -- 172.21.15.46:0/3719995412 learned_addr learned my addr 172.21.15.46:0/3719995412 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:24:56.864 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.020+0000 7f374db9f700 1 -- 172.21.15.46:0/3719995412 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f374810c190 con 0x7f3748109860 2024-04-26T21:24:56.864 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.021+0000 7f374db9f700 1 --2- 172.21.15.46:0/3719995412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3748109860 0x7f374810bc50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f37440098a0 tx=0x7f3744009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22054d9d0c406c94 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:56.864 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.021+0000 7f374cb9d700 1 -- 172.21.15.46:0/3719995412 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3744004160 con 0x7f3748109860 2024-04-26T21:24:56.864 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.021+0000 7f374cb9d700 1 -- 172.21.15.46:0/3719995412 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3744004300 con 0x7f3748109860 2024-04-26T21:24:56.864 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.021+0000 7f374cb9d700 1 -- 172.21.15.46:0/3719995412 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3744011af0 con 0x7f3748109860 2024-04-26T21:24:56.865 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.021+0000 7f374fe03700 1 -- 172.21.15.46:0/3719995412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3748109860 msgr2=0x7f374810bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:56.865 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.021+0000 7f374fe03700 1 --2- 172.21.15.46:0/3719995412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3748109860 0x7f374810bc50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f37440098a0 tx=0x7f3744009bb0 comp rx=0 tx=0).stop 2024-04-26T21:24:56.865 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.021+0000 7f374fe03700 1 -- 172.21.15.46:0/3719995412 shutdown_connections 2024-04-26T21:24:56.865 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.021+0000 7f374fe03700 1 --2- 172.21.15.46:0/3719995412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3748109860 0x7f374810bc50 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:56.865 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.021+0000 7f374fe03700 1 -- 172.21.15.46:0/3719995412 >> 172.21.15.46:0/3719995412 conn(0x7f37481031f0 msgr2=0x7f3748105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:24:56.865 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.022+0000 7f374fe03700 1 -- 172.21.15.46:0/3719995412 shutdown_connections 2024-04-26T21:24:56.865 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.022+0000 7f374fe03700 1 -- 172.21.15.46:0/3719995412 wait complete. 2024-04-26T21:24:56.865 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.022+0000 7f374fe03700 1 Processor -- start 2024-04-26T21:24:56.865 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.022+0000 7f374fe03700 1 -- start start 2024-04-26T21:24:56.865 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.022+0000 7f374fe03700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3748109860 0x7f374819cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:56.865 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.022+0000 7f374fe03700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f374810d100 con 0x7f3748109860 2024-04-26T21:24:56.865 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.023+0000 7f374db9f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3748109860 0x7f374819cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:56.865 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.023+0000 7f374db9f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3748109860 0x7f374819cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42830/0 (socket says 172.21.15.46:42830) 2024-04-26T21:24:56.865 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.023+0000 7f374db9f700 1 -- 172.21.15.46:0/2726941025 learned_addr learned my addr 172.21.15.46:0/2726941025 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:24:56.866 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.023+0000 7f374db9f700 1 -- 172.21.15.46:0/2726941025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f374819d220 con 0x7f3748109860 2024-04-26T21:24:56.866 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.023+0000 7f374db9f700 1 --2- 172.21.15.46:0/2726941025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3748109860 0x7f374819cce0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3744011ac0 tx=0x7f3744011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:56.866 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.023+0000 7f373effd700 1 -- 172.21.15.46:0/2726941025 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3744003970 con 0x7f3748109860 2024-04-26T21:24:56.866 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.023+0000 7f373effd700 1 -- 172.21.15.46:0/2726941025 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3744003b10 con 0x7f3748109860 2024-04-26T21:24:56.866 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.023+0000 7f374fe03700 1 -- 172.21.15.46:0/2726941025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f374819d4b0 con 0x7f3748109860 2024-04-26T21:24:56.866 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.023+0000 7f373effd700 1 -- 172.21.15.46:0/2726941025 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3744024610 con 0x7f3748109860 2024-04-26T21:24:56.866 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.023+0000 7f374fe03700 1 -- 172.21.15.46:0/2726941025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37481a0190 con 0x7f3748109860 2024-04-26T21:24:56.866 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.024+0000 7f373effd700 1 -- 172.21.15.46:0/2726941025 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f3744024840 con 0x7f3748109860 2024-04-26T21:24:56.866 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.024+0000 7f373effd700 1 --2- 172.21.15.46:0/2726941025 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7f3734039600 0x7f373403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:56.866 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.024+0000 7f373effd700 1 -- 172.21.15.46:0/2726941025 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f374404d8d0 con 0x7f3748109860 2024-04-26T21:24:56.866 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.024+0000 7f374d39e700 1 -- 172.21.15.46:0/2726941025 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7f3734039600 msgr2=0x7f373403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2896098005 2024-04-26T21:24:56.866 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.024+0000 7f374d39e700 1 --2- 172.21.15.46:0/2726941025 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7f3734039600 0x7f373403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T21:24:56.867 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.024+0000 7f374fe03700 1 -- 172.21.15.46:0/2726941025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3748064a80 con 0x7f3748109860 2024-04-26T21:24:56.867 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.030+0000 7f373effd700 1 -- 172.21.15.46:0/2726941025 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f37440172b0 con 0x7f3748109860 2024-04-26T21:24:56.867 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.225+0000 7f374d39e700 1 -- 172.21.15.46:0/2726941025 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7f3734039600 msgr2=0x7f373403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2896098005 2024-04-26T21:24:56.867 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.225+0000 7f374d39e700 1 --2- 172.21.15.46:0/2726941025 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7f3734039600 0x7f373403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T21:24:56.867 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.344+0000 7f374fe03700 1 -- 172.21.15.46:0/2726941025 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f37481a06e0 con 0x7f3748109860 2024-04-26T21:24:56.867 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.345+0000 7f373effd700 1 -- 172.21.15.46:0/2726941025 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f374404d2f0 con 0x7f3748109860 2024-04-26T21:24:56.867 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.347+0000 7f374fe03700 1 -- 172.21.15.46:0/2726941025 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7f3734039600 msgr2=0x7f373403bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T21:24:56.867 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.347+0000 7f374fe03700 1 --2- 172.21.15.46:0/2726941025 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7f3734039600 0x7f373403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:56.867 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.347+0000 7f374fe03700 1 -- 172.21.15.46:0/2726941025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3748109860 msgr2=0x7f374819cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:56.867 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.347+0000 7f374fe03700 1 --2- 172.21.15.46:0/2726941025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3748109860 0x7f374819cce0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3744011ac0 tx=0x7f3744011ba0 comp rx=0 tx=0).stop 2024-04-26T21:24:56.867 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.347+0000 7f374fe03700 1 -- 172.21.15.46:0/2726941025 shutdown_connections 2024-04-26T21:24:56.867 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.347+0000 7f374fe03700 1 --2- 172.21.15.46:0/2726941025 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7f3734039600 0x7f373403bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:56.867 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.347+0000 7f374fe03700 1 --2- 172.21.15.46:0/2726941025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3748109860 0x7f374819cce0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:56.867 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.347+0000 7f374fe03700 1 -- 172.21.15.46:0/2726941025 >> 172.21.15.46:0/2726941025 conn(0x7f37481031f0 msgr2=0x7f3748103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:24:56.868 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.347+0000 7f374fe03700 1 -- 172.21.15.46:0/2726941025 shutdown_connections 2024-04-26T21:24:56.868 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:56.347+0000 7f374fe03700 1 -- 172.21.15.46:0/2726941025 wait complete. 2024-04-26T21:24:56.868 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for the mgr to restart... 2024-04-26T21:24:56.868 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mgr epoch 5... 2024-04-26T21:24:57.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:24:56 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2726941025' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T21:24:57.304 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:57 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:57.127+0000 7f4be1e05200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T21:24:57.785 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:57 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:57.306+0000 7f4be1e05200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T21:24:57.785 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:57 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:57.485+0000 7f4be1e05200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T21:24:57.785 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:57 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:57.783+0000 7f4be1e05200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T21:24:58.053 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:57 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:57.880+0000 7f4be1e05200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T21:24:58.553 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:58 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:58.090+0000 7f4be1e05200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T21:24:58.553 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:58 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:58.291+0000 7f4be1e05200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T21:24:59.053 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:58 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:58.919+0000 7f4be1e05200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T21:24:59.669 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:59 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:59.357+0000 7f4be1e05200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T21:24:59.669 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:59 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:59.458+0000 7f4be1e05200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T21:24:59.669 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:24:59 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:24:59.668+0000 7f4be1e05200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T21:25:00.554 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:00 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:00.210+0000 7f4be1e05200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T21:25:00.554 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:00 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:00.305+0000 7f4be1e05200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T21:25:00.917 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:00 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:00.559+0000 7f4be1e05200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T21:25:00.917 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:00 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:00.697+0000 7f4be1e05200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T21:25:00.917 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:00 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:00.796+0000 7f4be1e05200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T21:25:00.917 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:00 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:00.915+0000 7f4be1e05200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T21:25:01.553 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:01 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:01.226+0000 7f4be1e05200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T21:25:02.130 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:01 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:01.852+0000 7f4be1e05200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T21:25:02.130 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:02 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:02.076+0000 7f4be1e05200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T21:25:02.541 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:02 smithi046 ceph-mon[29201]: Active manager daemon x restarted 2024-04-26T21:25:02.541 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:02 smithi046 ceph-mon[29201]: Activating manager daemon x 2024-04-26T21:25:02.541 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:02 smithi046 ceph-mon[29201]: osdmap e2: 0 total, 0 up, 0 in 2024-04-26T21:25:02.541 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:02 smithi046 ceph-mon[29201]: mgrmap e6: x(active, starting, since 0.0033757s) 2024-04-26T21:25:02.541 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:02 smithi046 ceph-mon[29201]: from='mgr.14118 172.21.15.46:0/1110106616' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:25:02.541 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:02 smithi046 ceph-mon[29201]: from='mgr.14118 172.21.15.46:0/1110106616' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T21:25:02.541 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:02 smithi046 ceph-mon[29201]: from='mgr.14118 172.21.15.46:0/1110106616' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T21:25:02.541 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:02 smithi046 ceph-mon[29201]: from='mgr.14118 172.21.15.46:0/1110106616' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T21:25:02.541 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:02 smithi046 ceph-mon[29201]: from='mgr.14118 172.21.15.46:0/1110106616' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T21:25:02.541 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:02 smithi046 ceph-mon[29201]: Manager daemon x is now available 2024-04-26T21:25:02.542 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:02 smithi046 ceph-mon[29201]: from='mgr.14118 172.21.15.46:0/1110106616' entity='mgr.x' 2024-04-26T21:25:02.542 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:02 smithi046 ceph-mon[29201]: from='mgr.14118 172.21.15.46:0/1110106616' entity='mgr.x' 2024-04-26T21:25:02.542 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:02 smithi046 ceph-mon[29201]: from='mgr.14118 172.21.15.46:0/1110106616' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:02.542 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:02 smithi046 ceph-mon[29201]: from='mgr.14118 172.21.15.46:0/1110106616' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:03.352 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:03 smithi046 ceph-mon[29201]: Found migration_current of "None". Setting to last migration. 2024-04-26T21:25:03.352 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:03 smithi046 ceph-mon[29201]: from='mgr.14118 172.21.15.46:0/1110106616' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T21:25:03.352 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:03 smithi046 ceph-mon[29201]: from='mgr.14118 172.21.15.46:0/1110106616' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T21:25:03.352 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:03 smithi046 ceph-mon[29201]: from='mgr.14118 172.21.15.46:0/1110106616' entity='mgr.x' 2024-04-26T21:25:03.352 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:03 smithi046 ceph-mon[29201]: from='mgr.14118 172.21.15.46:0/1110106616' entity='mgr.x' 2024-04-26T21:25:03.353 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:03 smithi046 ceph-mon[29201]: mgrmap e7: x(active, since 1.00663s) 2024-04-26T21:25:03.663 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-04-26T21:25:03.663 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-26T21:25:03.664 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T21:25:03.664 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-04-26T21:25:03.664 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.310+0000 7fe021e25700 1 Processor -- start 2024-04-26T21:25:03.664 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.310+0000 7fe021e25700 1 -- start start 2024-04-26T21:25:03.664 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.310+0000 7fe021e25700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe01c109820 0x7fe01c10bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:03.664 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.310+0000 7fe021e25700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe01c07d6d0 con 0x7fe01c109820 2024-04-26T21:25:03.664 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.310+0000 7fe01b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe01c109820 0x7fe01c10bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:03.664 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.310+0000 7fe01b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe01c109820 0x7fe01c10bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42846/0 (socket says 172.21.15.46:42846) 2024-04-26T21:25:03.664 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.310+0000 7fe01b7fe700 1 -- 172.21.15.46:0/1350953836 learned_addr learned my addr 172.21.15.46:0/1350953836 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:03.664 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.310+0000 7fe01b7fe700 1 -- 172.21.15.46:0/1350953836 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe01c10c150 con 0x7fe01c109820 2024-04-26T21:25:03.664 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.311+0000 7fe01b7fe700 1 --2- 172.21.15.46:0/1350953836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe01c109820 0x7fe01c10bc10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe00c0098a0 tx=0x7fe00c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffafcd6f5ef54923 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:03.664 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.311+0000 7fe01a7fc700 1 -- 172.21.15.46:0/1350953836 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe00c004160 con 0x7fe01c109820 2024-04-26T21:25:03.664 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.311+0000 7fe01a7fc700 1 -- 172.21.15.46:0/1350953836 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe00c004300 con 0x7fe01c109820 2024-04-26T21:25:03.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.311+0000 7fe01a7fc700 1 -- 172.21.15.46:0/1350953836 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe00c011af0 con 0x7fe01c109820 2024-04-26T21:25:03.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.311+0000 7fe021e25700 1 -- 172.21.15.46:0/1350953836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe01c109820 msgr2=0x7fe01c10bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:03.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.311+0000 7fe021e25700 1 --2- 172.21.15.46:0/1350953836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe01c109820 0x7fe01c10bc10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe00c0098a0 tx=0x7fe00c009bb0 comp rx=0 tx=0).stop 2024-04-26T21:25:03.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.311+0000 7fe021e25700 1 -- 172.21.15.46:0/1350953836 shutdown_connections 2024-04-26T21:25:03.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.311+0000 7fe021e25700 1 --2- 172.21.15.46:0/1350953836 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe01c109820 0x7fe01c10bc10 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:03.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.312+0000 7fe021e25700 1 -- 172.21.15.46:0/1350953836 >> 172.21.15.46:0/1350953836 conn(0x7fe01c1031f0 msgr2=0x7fe01c105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:03.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.312+0000 7fe021e25700 1 -- 172.21.15.46:0/1350953836 shutdown_connections 2024-04-26T21:25:03.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.312+0000 7fe021e25700 1 -- 172.21.15.46:0/1350953836 wait complete. 2024-04-26T21:25:03.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.312+0000 7fe021e25700 1 Processor -- start 2024-04-26T21:25:03.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.312+0000 7fe021e25700 1 -- start start 2024-04-26T21:25:03.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.313+0000 7fe021e25700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe01c109820 0x7fe01c19ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:03.665 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.313+0000 7fe021e25700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe01c10d0c0 con 0x7fe01c109820 2024-04-26T21:25:03.666 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.313+0000 7fe01b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe01c109820 0x7fe01c19ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:03.666 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.313+0000 7fe01b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe01c109820 0x7fe01c19ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42850/0 (socket says 172.21.15.46:42850) 2024-04-26T21:25:03.666 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.313+0000 7fe01b7fe700 1 -- 172.21.15.46:0/2024908650 learned_addr learned my addr 172.21.15.46:0/2024908650 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:03.666 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.313+0000 7fe01b7fe700 1 -- 172.21.15.46:0/2024908650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe01c19f440 con 0x7fe01c109820 2024-04-26T21:25:03.666 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.313+0000 7fe01b7fe700 1 --2- 172.21.15.46:0/2024908650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe01c109820 0x7fe01c19ef00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe00c009fb0 tx=0x7fe00c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:03.666 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.313+0000 7fe018ff9700 1 -- 172.21.15.46:0/2024908650 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe00c0036a0 con 0x7fe01c109820 2024-04-26T21:25:03.666 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.313+0000 7fe018ff9700 1 -- 172.21.15.46:0/2024908650 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe00c01a430 con 0x7fe01c109820 2024-04-26T21:25:03.666 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.313+0000 7fe021e25700 1 -- 172.21.15.46:0/2024908650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe01c19f6d0 con 0x7fe01c109820 2024-04-26T21:25:03.666 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.313+0000 7fe018ff9700 1 -- 172.21.15.46:0/2024908650 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe00c019550 con 0x7fe01c109820 2024-04-26T21:25:03.666 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.313+0000 7fe021e25700 1 -- 172.21.15.46:0/2024908650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe01c1a23c0 con 0x7fe01c109820 2024-04-26T21:25:03.666 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.314+0000 7fe018ff9700 1 -- 172.21.15.46:0/2024908650 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fe00c019800 con 0x7fe01c109820 2024-04-26T21:25:03.666 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.315+0000 7fe018ff9700 1 --2- 172.21.15.46:0/2024908650 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7fe0040395b0 0x7fe00403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:03.666 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.315+0000 7fe018ff9700 1 -- 172.21.15.46:0/2024908650 --> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe00403c180 con 0x7fe0040395b0 2024-04-26T21:25:03.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.315+0000 7fe018ff9700 1 -- 172.21.15.46:0/2024908650 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe00c04d8e0 con 0x7fe01c109820 2024-04-26T21:25:03.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.315+0000 7fe01affd700 1 -- 172.21.15.46:0/2024908650 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7fe0040395b0 msgr2=0x7fe00403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2896098005 2024-04-26T21:25:03.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.315+0000 7fe01affd700 1 --2- 172.21.15.46:0/2024908650 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7fe0040395b0 0x7fe00403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T21:25:03.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.516+0000 7fe01affd700 1 -- 172.21.15.46:0/2024908650 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7fe0040395b0 msgr2=0x7fe00403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2896098005 2024-04-26T21:25:03.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.516+0000 7fe01affd700 1 --2- 172.21.15.46:0/2024908650 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7fe0040395b0 0x7fe00403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T21:25:03.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.917+0000 7fe01affd700 1 -- 172.21.15.46:0/2024908650 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7fe0040395b0 msgr2=0x7fe00403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2896098005 2024-04-26T21:25:03.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:57.917+0000 7fe01affd700 1 --2- 172.21.15.46:0/2024908650 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7fe0040395b0 0x7fe00403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T21:25:03.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:58.718+0000 7fe01affd700 1 -- 172.21.15.46:0/2024908650 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7fe0040395b0 msgr2=0x7fe00403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2896098005 2024-04-26T21:25:03.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:24:58.718+0000 7fe01affd700 1 --2- 172.21.15.46:0/2024908650 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7fe0040395b0 0x7fe00403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T21:25:03.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:00.319+0000 7fe01affd700 1 -- 172.21.15.46:0/2024908650 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7fe0040395b0 msgr2=0x7fe00403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2896098005 2024-04-26T21:25:03.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:00.319+0000 7fe01affd700 1 --2- 172.21.15.46:0/2024908650 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7fe0040395b0 0x7fe00403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T21:25:03.667 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:02.080+0000 7fe018ff9700 1 -- 172.21.15.46:0/2024908650 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7fe00c019b40 con 0x7fe01c109820 2024-04-26T21:25:03.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:02.080+0000 7fe018ff9700 1 -- 172.21.15.46:0/2024908650 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7fe0040395b0 msgr2=0x7fe00403ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T21:25:03.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:02.080+0000 7fe018ff9700 1 --2- 172.21.15.46:0/2024908650 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7fe0040395b0 0x7fe00403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:03.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:03.081+0000 7fe018ff9700 1 -- 172.21.15.46:0/2024908650 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7fe00c04f720 con 0x7fe01c109820 2024-04-26T21:25:03.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:03.081+0000 7fe018ff9700 1 --2- 172.21.15.46:0/2024908650 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fe00403cea0 0x7fe00403f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:03.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:03.081+0000 7fe018ff9700 1 -- 172.21.15.46:0/2024908650 --> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe00c001d30 con 0x7fe00403cea0 2024-04-26T21:25:03.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:03.082+0000 7fe01affd700 1 --2- 172.21.15.46:0/2024908650 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fe00403cea0 0x7fe00403f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:03.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:03.082+0000 7fe01affd700 1 --2- 172.21.15.46:0/2024908650 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fe00403cea0 0x7fe00403f290 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe010003a10 tx=0x7fe0100092b0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:03.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:03.083+0000 7fe018ff9700 1 -- 172.21.15.46:0/2024908650 <== mgr.14118 v2:172.21.15.46:6800/2053782984 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7fe00c001d30 con 0x7fe00403cea0 2024-04-26T21:25:03.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:03.092+0000 7fe021e25700 1 -- 172.21.15.46:0/2024908650 --> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fe01c06a540 con 0x7fe00403cea0 2024-04-26T21:25:03.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:03.092+0000 7fe018ff9700 1 -- 172.21.15.46:0/2024908650 <== mgr.14118 v2:172.21.15.46:6800/2053782984 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fe01c06a540 con 0x7fe00403cea0 2024-04-26T21:25:03.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:03.092+0000 7fe021e25700 1 -- 172.21.15.46:0/2024908650 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fe00403cea0 msgr2=0x7fe00403f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:03.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:03.092+0000 7fe021e25700 1 --2- 172.21.15.46:0/2024908650 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fe00403cea0 0x7fe00403f290 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe010003a10 tx=0x7fe0100092b0 comp rx=0 tx=0).stop 2024-04-26T21:25:03.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:03.092+0000 7fe021e25700 1 -- 172.21.15.46:0/2024908650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe01c109820 msgr2=0x7fe01c19ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:03.668 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:03.092+0000 7fe021e25700 1 --2- 172.21.15.46:0/2024908650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe01c109820 0x7fe01c19ef00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe00c009fb0 tx=0x7fe00c011940 comp rx=0 tx=0).stop 2024-04-26T21:25:03.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:03.092+0000 7fe021e25700 1 -- 172.21.15.46:0/2024908650 shutdown_connections 2024-04-26T21:25:03.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:03.092+0000 7fe021e25700 1 --2- 172.21.15.46:0/2024908650 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fe00403cea0 0x7fe00403f290 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:03.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:03.092+0000 7fe021e25700 1 --2- 172.21.15.46:0/2024908650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe01c109820 0x7fe01c19ef00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:03.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:03.092+0000 7fe021e25700 1 --2- 172.21.15.46:0/2024908650 >> [v2:172.21.15.46:6800/2896098005,v1:172.21.15.46:6801/2896098005] conn(0x7fe0040395b0 0x7fe00403ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:03.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:03.092+0000 7fe021e25700 1 -- 172.21.15.46:0/2024908650 >> 172.21.15.46:0/2024908650 conn(0x7fe01c1031f0 msgr2=0x7fe01c103d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:03.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:03.093+0000 7fe021e25700 1 -- 172.21.15.46:0/2024908650 shutdown_connections 2024-04-26T21:25:03.669 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:03.093+0000 7fe021e25700 1 -- 172.21.15.46:0/2024908650 wait complete. 2024-04-26T21:25:03.669 INFO:teuthology.orchestra.run.smithi046.stdout:mgr epoch 5 is available 2024-04-26T21:25:03.669 INFO:teuthology.orchestra.run.smithi046.stdout:Setting orchestrator backend to cephadm... 2024-04-26T21:25:04.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:04 smithi046 ceph-mon[29201]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T21:25:04.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:04 smithi046 ceph-mon[29201]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T21:25:04.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:04 smithi046 ceph-mon[29201]: [26/Apr/2024:21:25:03] ENGINE Bus STARTING 2024-04-26T21:25:04.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:04 smithi046 ceph-mon[29201]: [26/Apr/2024:21:25:03] ENGINE Serving on http://172.21.15.46:8765 2024-04-26T21:25:04.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:04 smithi046 ceph-mon[29201]: [26/Apr/2024:21:25:03] ENGINE Serving on https://172.21.15.46:7150 2024-04-26T21:25:04.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:04 smithi046 ceph-mon[29201]: [26/Apr/2024:21:25:03] ENGINE Bus STARTED 2024-04-26T21:25:04.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:04 smithi046 ceph-mon[29201]: from='mgr.14118 172.21.15.46:0/1110106616' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:04.892 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.103+0000 7f8df2445700 1 Processor -- start 2024-04-26T21:25:04.893 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.103+0000 7f8df2445700 1 -- start start 2024-04-26T21:25:04.893 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.104+0000 7f8df2445700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dec108dd0 0x7f8dec1091f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:04.893 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.104+0000 7f8df2445700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8dec1097c0 con 0x7f8dec108dd0 2024-04-26T21:25:04.893 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.104+0000 7f8debfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dec108dd0 0x7f8dec1091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:04.893 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.104+0000 7f8debfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dec108dd0 0x7f8dec1091f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42898/0 (socket says 172.21.15.46:42898) 2024-04-26T21:25:04.893 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.104+0000 7f8debfff700 1 -- 172.21.15.46:0/3149040108 learned_addr learned my addr 172.21.15.46:0/3149040108 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:04.893 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.104+0000 7f8debfff700 1 -- 172.21.15.46:0/3149040108 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dec109f80 con 0x7f8dec108dd0 2024-04-26T21:25:04.893 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.104+0000 7f8debfff700 1 --2- 172.21.15.46:0/3149040108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dec108dd0 0x7f8dec1091f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc0098a0 tx=0x7f8ddc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b01d437fe20c003 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:04.893 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.104+0000 7f8deaffd700 1 -- 172.21.15.46:0/3149040108 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ddc004160 con 0x7f8dec108dd0 2024-04-26T21:25:04.893 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.104+0000 7f8deaffd700 1 -- 172.21.15.46:0/3149040108 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8ddc004300 con 0x7f8dec108dd0 2024-04-26T21:25:04.893 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.104+0000 7f8deaffd700 1 -- 172.21.15.46:0/3149040108 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ddc011af0 con 0x7f8dec108dd0 2024-04-26T21:25:04.893 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.105+0000 7f8df2445700 1 -- 172.21.15.46:0/3149040108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dec108dd0 msgr2=0x7f8dec1091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:04.893 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.105+0000 7f8df2445700 1 --2- 172.21.15.46:0/3149040108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dec108dd0 0x7f8dec1091f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc0098a0 tx=0x7f8ddc009bb0 comp rx=0 tx=0).stop 2024-04-26T21:25:04.894 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.105+0000 7f8df2445700 1 -- 172.21.15.46:0/3149040108 shutdown_connections 2024-04-26T21:25:04.894 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.105+0000 7f8df2445700 1 --2- 172.21.15.46:0/3149040108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dec108dd0 0x7f8dec1091f0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:04.894 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.105+0000 7f8df2445700 1 -- 172.21.15.46:0/3149040108 >> 172.21.15.46:0/3149040108 conn(0x7f8dec104370 msgr2=0x7f8dec1067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:04.894 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.105+0000 7f8df2445700 1 -- 172.21.15.46:0/3149040108 shutdown_connections 2024-04-26T21:25:04.894 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.105+0000 7f8df2445700 1 -- 172.21.15.46:0/3149040108 wait complete. 2024-04-26T21:25:04.894 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.106+0000 7f8df2445700 1 Processor -- start 2024-04-26T21:25:04.894 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.106+0000 7f8df2445700 1 -- start start 2024-04-26T21:25:04.894 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.106+0000 7f8df2445700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dec108dd0 0x7f8dec19cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:04.894 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.106+0000 7f8df2445700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8dec10d330 con 0x7f8dec108dd0 2024-04-26T21:25:04.894 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.106+0000 7f8debfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dec108dd0 0x7f8dec19cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:04.894 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.106+0000 7f8debfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dec108dd0 0x7f8dec19cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42906/0 (socket says 172.21.15.46:42906) 2024-04-26T21:25:04.894 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.106+0000 7f8debfff700 1 -- 172.21.15.46:0/1882892839 learned_addr learned my addr 172.21.15.46:0/1882892839 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:04.894 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.106+0000 7f8debfff700 1 -- 172.21.15.46:0/1882892839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dec19d250 con 0x7f8dec108dd0 2024-04-26T21:25:04.895 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.107+0000 7f8debfff700 1 --2- 172.21.15.46:0/1882892839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dec108dd0 0x7f8dec19cd10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc0117b0 tx=0x7f8ddc011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:04.895 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.107+0000 7f8de97fa700 1 -- 172.21.15.46:0/1882892839 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ddc003af0 con 0x7f8dec108dd0 2024-04-26T21:25:04.895 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.107+0000 7f8de97fa700 1 -- 172.21.15.46:0/1882892839 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8ddc020070 con 0x7f8dec108dd0 2024-04-26T21:25:04.895 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.107+0000 7f8de97fa700 1 -- 172.21.15.46:0/1882892839 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8ddc019630 con 0x7f8dec108dd0 2024-04-26T21:25:04.895 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.107+0000 7f8df2445700 1 -- 172.21.15.46:0/1882892839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dec19d4e0 con 0x7f8dec108dd0 2024-04-26T21:25:04.895 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.107+0000 7f8df2445700 1 -- 172.21.15.46:0/1882892839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dec1a01c0 con 0x7f8dec108dd0 2024-04-26T21:25:04.895 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.108+0000 7f8de97fa700 1 -- 172.21.15.46:0/1882892839 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f8ddc01a9c0 con 0x7f8dec108dd0 2024-04-26T21:25:04.895 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.108+0000 7f8df2445700 1 -- 172.21.15.46:0/1882892839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dec064a80 con 0x7f8dec108dd0 2024-04-26T21:25:04.895 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.108+0000 7f8de97fa700 1 --2- 172.21.15.46:0/1882892839 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f8dd4039490 0x7f8dd403b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:04.895 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.108+0000 7f8de97fa700 1 -- 172.21.15.46:0/1882892839 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8ddc04d980 con 0x7f8dec108dd0 2024-04-26T21:25:04.895 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.108+0000 7f8deb7fe700 1 --2- 172.21.15.46:0/1882892839 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f8dd4039490 0x7f8dd403b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:04.895 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.108+0000 7f8deb7fe700 1 --2- 172.21.15.46:0/1882892839 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f8dd4039490 0x7f8dd403b950 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8de0006fd0 tx=0x7f8de0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:04.895 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.113+0000 7f8de97fa700 1 -- 172.21.15.46:0/1882892839 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8ddc00e260 con 0x7f8dec108dd0 2024-04-26T21:25:04.895 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.132+0000 7f8de97fa700 1 -- 172.21.15.46:0/1882892839 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f8ddc01a430 con 0x7f8dec108dd0 2024-04-26T21:25:04.896 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.338+0000 7f8df2445700 1 -- 172.21.15.46:0/1882892839 --> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f8dec105630 con 0x7f8dd4039490 2024-04-26T21:25:04.896 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.343+0000 7f8de97fa700 1 -- 172.21.15.46:0/1882892839 <== mgr.14118 v2:172.21.15.46:6800/2053782984 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f8dec105630 con 0x7f8dd4039490 2024-04-26T21:25:04.896 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.345+0000 7f8df2445700 1 -- 172.21.15.46:0/1882892839 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f8dd4039490 msgr2=0x7f8dd403b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:04.896 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.345+0000 7f8df2445700 1 --2- 172.21.15.46:0/1882892839 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f8dd4039490 0x7f8dd403b950 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8de0006fd0 tx=0x7f8de0006e40 comp rx=0 tx=0).stop 2024-04-26T21:25:04.896 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.345+0000 7f8df2445700 1 -- 172.21.15.46:0/1882892839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dec108dd0 msgr2=0x7f8dec19cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:04.896 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.345+0000 7f8df2445700 1 --2- 172.21.15.46:0/1882892839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dec108dd0 0x7f8dec19cd10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc0117b0 tx=0x7f8ddc011b70 comp rx=0 tx=0).stop 2024-04-26T21:25:04.896 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.346+0000 7f8df2445700 1 -- 172.21.15.46:0/1882892839 shutdown_connections 2024-04-26T21:25:04.896 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.346+0000 7f8df2445700 1 --2- 172.21.15.46:0/1882892839 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f8dd4039490 0x7f8dd403b950 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:04.896 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.346+0000 7f8df2445700 1 --2- 172.21.15.46:0/1882892839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dec108dd0 0x7f8dec19cd10 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:04.896 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.346+0000 7f8df2445700 1 -- 172.21.15.46:0/1882892839 >> 172.21.15.46:0/1882892839 conn(0x7f8dec104370 msgr2=0x7f8dec104f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:04.896 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.346+0000 7f8df2445700 1 -- 172.21.15.46:0/1882892839 shutdown_connections 2024-04-26T21:25:04.896 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:04.346+0000 7f8df2445700 1 -- 172.21.15.46:0/1882892839 wait complete. 2024-04-26T21:25:05.543 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:05 smithi046 ceph-mon[29201]: mgrmap e8: x(active, since 2s) 2024-04-26T21:25:05.543 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:05 smithi046 ceph-mon[29201]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:25:05.543 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:05 smithi046 ceph-mon[29201]: from='mgr.14118 172.21.15.46:0/1110106616' entity='mgr.x' 2024-04-26T21:25:05.543 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:05 smithi046 ceph-mon[29201]: from='mgr.14118 172.21.15.46:0/1110106616' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:06.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-26T21:25:06.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.322+0000 7f6e6141f700 1 Processor -- start 2024-04-26T21:25:06.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.322+0000 7f6e6141f700 1 -- start start 2024-04-26T21:25:06.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.323+0000 7f6e6141f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e5c107c50 0x7f6e5c108070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:06.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.323+0000 7f6e6141f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5c108640 con 0x7f6e5c107c50 2024-04-26T21:25:06.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.323+0000 7f6e5affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e5c107c50 0x7f6e5c108070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:06.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.323+0000 7f6e5affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e5c107c50 0x7f6e5c108070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42912/0 (socket says 172.21.15.46:42912) 2024-04-26T21:25:06.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.323+0000 7f6e5affd700 1 -- 172.21.15.46:0/1175115261 learned_addr learned my addr 172.21.15.46:0/1175115261 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:06.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.323+0000 7f6e5affd700 1 -- 172.21.15.46:0/1175115261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e5c108e50 con 0x7f6e5c107c50 2024-04-26T21:25:06.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.323+0000 7f6e5affd700 1 --2- 172.21.15.46:0/1175115261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e5c107c50 0x7f6e5c108070 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6e54009e80 tx=0x7f6e5400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74c8bf81fc08a19d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:06.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.323+0000 7f6e59ffb700 1 -- 172.21.15.46:0/1175115261 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6e54017070 con 0x7f6e5c107c50 2024-04-26T21:25:06.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.324+0000 7f6e59ffb700 1 -- 172.21.15.46:0/1175115261 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6e5400b8c0 con 0x7f6e5c107c50 2024-04-26T21:25:06.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.324+0000 7f6e59ffb700 1 -- 172.21.15.46:0/1175115261 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6e54004490 con 0x7f6e5c107c50 2024-04-26T21:25:06.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.324+0000 7f6e6141f700 1 -- 172.21.15.46:0/1175115261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e5c107c50 msgr2=0x7f6e5c108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:06.074 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.324+0000 7f6e6141f700 1 --2- 172.21.15.46:0/1175115261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e5c107c50 0x7f6e5c108070 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6e54009e80 tx=0x7f6e5400b330 comp rx=0 tx=0).stop 2024-04-26T21:25:06.074 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.324+0000 7f6e6141f700 1 -- 172.21.15.46:0/1175115261 shutdown_connections 2024-04-26T21:25:06.074 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.324+0000 7f6e6141f700 1 --2- 172.21.15.46:0/1175115261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e5c107c50 0x7f6e5c108070 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:06.074 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.324+0000 7f6e6141f700 1 -- 172.21.15.46:0/1175115261 >> 172.21.15.46:0/1175115261 conn(0x7f6e5c1031f0 msgr2=0x7f6e5c105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:06.074 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.324+0000 7f6e6141f700 1 -- 172.21.15.46:0/1175115261 shutdown_connections 2024-04-26T21:25:06.074 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.324+0000 7f6e6141f700 1 -- 172.21.15.46:0/1175115261 wait complete. 2024-04-26T21:25:06.074 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.325+0000 7f6e6141f700 1 Processor -- start 2024-04-26T21:25:06.074 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.325+0000 7f6e6141f700 1 -- start start 2024-04-26T21:25:06.074 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.325+0000 7f6e6141f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e5c107c50 0x7f6e5c19ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:06.074 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.325+0000 7f6e6141f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5c10c200 con 0x7f6e5c107c50 2024-04-26T21:25:06.074 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.325+0000 7f6e5affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e5c107c50 0x7f6e5c19ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:06.074 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.325+0000 7f6e5affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e5c107c50 0x7f6e5c19ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42916/0 (socket says 172.21.15.46:42916) 2024-04-26T21:25:06.074 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.325+0000 7f6e5affd700 1 -- 172.21.15.46:0/1869542637 learned_addr learned my addr 172.21.15.46:0/1869542637 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:06.074 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.326+0000 7f6e5affd700 1 -- 172.21.15.46:0/1869542637 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e5c19d210 con 0x7f6e5c107c50 2024-04-26T21:25:06.074 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.326+0000 7f6e5affd700 1 --2- 172.21.15.46:0/1869542637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e5c107c50 0x7f6e5c19ccd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6e5400b7b0 tx=0x7f6e5400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:06.075 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.326+0000 7f6e53fff700 1 -- 172.21.15.46:0/1869542637 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6e54020020 con 0x7f6e5c107c50 2024-04-26T21:25:06.075 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.326+0000 7f6e6141f700 1 -- 172.21.15.46:0/1869542637 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e5c19d4a0 con 0x7f6e5c107c50 2024-04-26T21:25:06.075 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.326+0000 7f6e53fff700 1 -- 172.21.15.46:0/1869542637 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6e54012430 con 0x7f6e5c107c50 2024-04-26T21:25:06.075 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.326+0000 7f6e53fff700 1 -- 172.21.15.46:0/1869542637 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6e54017040 con 0x7f6e5c107c50 2024-04-26T21:25:06.075 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.326+0000 7f6e6141f700 1 -- 172.21.15.46:0/1869542637 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e5c1a0180 con 0x7f6e5c107c50 2024-04-26T21:25:06.075 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.327+0000 7f6e53fff700 1 -- 172.21.15.46:0/1869542637 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f6e54023020 con 0x7f6e5c107c50 2024-04-26T21:25:06.075 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.327+0000 7f6e6141f700 1 -- 172.21.15.46:0/1869542637 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e5c06a540 con 0x7f6e5c107c50 2024-04-26T21:25:06.075 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.327+0000 7f6e53fff700 1 --2- 172.21.15.46:0/1869542637 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f6e48039560 0x7f6e4803ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:06.075 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.327+0000 7f6e53fff700 1 -- 172.21.15.46:0/1869542637 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6e5400f050 con 0x7f6e5c107c50 2024-04-26T21:25:06.075 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.327+0000 7f6e5a7fc700 1 --2- 172.21.15.46:0/1869542637 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f6e48039560 0x7f6e4803ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:06.075 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.328+0000 7f6e5a7fc700 1 --2- 172.21.15.46:0/1869542637 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f6e48039560 0x7f6e4803ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6e44006fd0 tx=0x7f6e44006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:06.075 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.333+0000 7f6e53fff700 1 -- 172.21.15.46:0/1869542637 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6e5402a020 con 0x7f6e5c107c50 2024-04-26T21:25:06.075 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.542+0000 7f6e6141f700 1 -- 172.21.15.46:0/1869542637 --> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f6e5c02e870 con 0x7f6e48039560 2024-04-26T21:25:06.075 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.542+0000 7f6e53fff700 1 -- 172.21.15.46:0/1869542637 <== mgr.14118 v2:172.21.15.46:6800/2053782984 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f6e5c02e870 con 0x7f6e48039560 2024-04-26T21:25:06.075 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.544+0000 7f6e6141f700 1 -- 172.21.15.46:0/1869542637 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f6e48039560 msgr2=0x7f6e4803ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:06.075 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.544+0000 7f6e6141f700 1 --2- 172.21.15.46:0/1869542637 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f6e48039560 0x7f6e4803ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6e44006fd0 tx=0x7f6e44006e40 comp rx=0 tx=0).stop 2024-04-26T21:25:06.076 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.544+0000 7f6e6141f700 1 -- 172.21.15.46:0/1869542637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e5c107c50 msgr2=0x7f6e5c19ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:06.076 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.544+0000 7f6e6141f700 1 --2- 172.21.15.46:0/1869542637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e5c107c50 0x7f6e5c19ccd0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6e5400b7b0 tx=0x7f6e5400be20 comp rx=0 tx=0).stop 2024-04-26T21:25:06.076 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.544+0000 7f6e6141f700 1 -- 172.21.15.46:0/1869542637 shutdown_connections 2024-04-26T21:25:06.076 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.544+0000 7f6e6141f700 1 --2- 172.21.15.46:0/1869542637 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f6e48039560 0x7f6e4803ba20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:06.076 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.544+0000 7f6e6141f700 1 --2- 172.21.15.46:0/1869542637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e5c107c50 0x7f6e5c19ccd0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:06.076 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.544+0000 7f6e6141f700 1 -- 172.21.15.46:0/1869542637 >> 172.21.15.46:0/1869542637 conn(0x7f6e5c1031f0 msgr2=0x7f6e5c104080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:06.076 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.545+0000 7f6e6141f700 1 -- 172.21.15.46:0/1869542637 shutdown_connections 2024-04-26T21:25:06.076 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:05.545+0000 7f6e6141f700 1 -- 172.21.15.46:0/1869542637 wait complete. 2024-04-26T21:25:06.076 INFO:teuthology.orchestra.run.smithi046.stdout:Generating ssh key... 2024-04-26T21:25:06.326 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:06 smithi046 ceph-mon[29201]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:25:06.992 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:06 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: Generating public/private rsa key pair. 2024-04-26T21:25:07.021 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:06 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: Your identification has been saved in /tmp/tmp1l61bfdh/key. 2024-04-26T21:25:07.021 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:06 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: Your public key has been saved in /tmp/tmp1l61bfdh/key.pub. 2024-04-26T21:25:07.022 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:06 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: The key fingerprint is: 2024-04-26T21:25:07.023 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:06 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: SHA256:2Vv7pM+KYCvHKYwJoelkYP3tVoUi06KbFCymv7TN9XM ceph-36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:25:07.023 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:06 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: The key's randomart image is: 2024-04-26T21:25:07.023 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:06 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: +---[RSA 3072]----+ 2024-04-26T21:25:07.024 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:06 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: | | 2024-04-26T21:25:07.024 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:06 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: | | 2024-04-26T21:25:07.024 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:06 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: | o . . | 2024-04-26T21:25:07.024 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:06 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: |.+.+ + o + . | 2024-04-26T21:25:07.024 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:06 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: |=o..+ = S o . | 2024-04-26T21:25:07.024 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:06 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: |+o.o . . . o . | 2024-04-26T21:25:07.024 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:06 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: |+.o.o+o.+.. . . | 2024-04-26T21:25:07.024 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:06 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: | o.*o.+==oE. = | 2024-04-26T21:25:07.025 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:06 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: | o.o .+oo. oo+ | 2024-04-26T21:25:07.025 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:06 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: +----[SHA256]-----+ 2024-04-26T21:25:07.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.520+0000 7f653e668700 1 Processor -- start 2024-04-26T21:25:07.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.520+0000 7f653e668700 1 -- start start 2024-04-26T21:25:07.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.521+0000 7f653e668700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6538109800 0x7f653810bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:07.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.521+0000 7f653e668700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f653807cea0 con 0x7f6538109800 2024-04-26T21:25:07.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.521+0000 7f6537fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6538109800 0x7f653810bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:07.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.521+0000 7f6537fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6538109800 0x7f653810bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57890/0 (socket says 172.21.15.46:57890) 2024-04-26T21:25:07.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.521+0000 7f6537fff700 1 -- 172.21.15.46:0/2332868634 learned_addr learned my addr 172.21.15.46:0/2332868634 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:07.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.521+0000 7f6537fff700 1 -- 172.21.15.46:0/2332868634 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f653810c130 con 0x7f6538109800 2024-04-26T21:25:07.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.521+0000 7f6537fff700 1 --2- 172.21.15.46:0/2332868634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6538109800 0x7f653810bbf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f65280098a0 tx=0x7f6528009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6169b76f416e7f40 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:07.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.521+0000 7f6536ffd700 1 -- 172.21.15.46:0/2332868634 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6528004160 con 0x7f6538109800 2024-04-26T21:25:07.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.522+0000 7f6536ffd700 1 -- 172.21.15.46:0/2332868634 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6528004300 con 0x7f6538109800 2024-04-26T21:25:07.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.522+0000 7f6536ffd700 1 -- 172.21.15.46:0/2332868634 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6528011af0 con 0x7f6538109800 2024-04-26T21:25:07.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.522+0000 7f653e668700 1 -- 172.21.15.46:0/2332868634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6538109800 msgr2=0x7f653810bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:07.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.522+0000 7f653e668700 1 --2- 172.21.15.46:0/2332868634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6538109800 0x7f653810bbf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f65280098a0 tx=0x7f6528009bb0 comp rx=0 tx=0).stop 2024-04-26T21:25:07.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.522+0000 7f653e668700 1 -- 172.21.15.46:0/2332868634 shutdown_connections 2024-04-26T21:25:07.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.522+0000 7f653e668700 1 --2- 172.21.15.46:0/2332868634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6538109800 0x7f653810bbf0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:07.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.522+0000 7f653e668700 1 -- 172.21.15.46:0/2332868634 >> 172.21.15.46:0/2332868634 conn(0x7f65381031f0 msgr2=0x7f6538105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:07.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.522+0000 7f653e668700 1 -- 172.21.15.46:0/2332868634 shutdown_connections 2024-04-26T21:25:07.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.522+0000 7f653e668700 1 -- 172.21.15.46:0/2332868634 wait complete. 2024-04-26T21:25:07.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.523+0000 7f653e668700 1 Processor -- start 2024-04-26T21:25:07.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.523+0000 7f653e668700 1 -- start start 2024-04-26T21:25:07.706 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.523+0000 7f653e668700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6538109800 0x7f6538109080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:07.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.523+0000 7f653e668700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f653810d100 con 0x7f6538109800 2024-04-26T21:25:07.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.524+0000 7f6537fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6538109800 0x7f6538109080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:07.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.524+0000 7f6537fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6538109800 0x7f6538109080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57900/0 (socket says 172.21.15.46:57900) 2024-04-26T21:25:07.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.524+0000 7f6537fff700 1 -- 172.21.15.46:0/1400825589 learned_addr learned my addr 172.21.15.46:0/1400825589 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:07.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.524+0000 7f6537fff700 1 -- 172.21.15.46:0/1400825589 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65381095c0 con 0x7f6538109800 2024-04-26T21:25:07.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.524+0000 7f6537fff700 1 --2- 172.21.15.46:0/1400825589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6538109800 0x7f6538109080 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6528011ac0 tx=0x7f6528011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:07.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.524+0000 7f65357fa700 1 -- 172.21.15.46:0/1400825589 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6528003970 con 0x7f6538109800 2024-04-26T21:25:07.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.524+0000 7f65357fa700 1 -- 172.21.15.46:0/1400825589 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6528003b10 con 0x7f6538109800 2024-04-26T21:25:07.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.524+0000 7f653e668700 1 -- 172.21.15.46:0/1400825589 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65381077b0 con 0x7f6538109800 2024-04-26T21:25:07.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.524+0000 7f65357fa700 1 -- 172.21.15.46:0/1400825589 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6528024610 con 0x7f6538109800 2024-04-26T21:25:07.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.524+0000 7f653e668700 1 -- 172.21.15.46:0/1400825589 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6538107cf0 con 0x7f6538109800 2024-04-26T21:25:07.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.525+0000 7f65357fa700 1 -- 172.21.15.46:0/1400825589 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f6528024840 con 0x7f6538109800 2024-04-26T21:25:07.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.525+0000 7f653e668700 1 -- 172.21.15.46:0/1400825589 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6538064a80 con 0x7f6538109800 2024-04-26T21:25:07.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.525+0000 7f65357fa700 1 --2- 172.21.15.46:0/1400825589 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f6524039600 0x7f652403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:07.707 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.525+0000 7f65357fa700 1 -- 172.21.15.46:0/1400825589 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f652804db10 con 0x7f6538109800 2024-04-26T21:25:07.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.526+0000 7f65377fe700 1 --2- 172.21.15.46:0/1400825589 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f6524039600 0x7f652403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:07.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.526+0000 7f65377fe700 1 --2- 172.21.15.46:0/1400825589 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f6524039600 0x7f652403bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6520006fd0 tx=0x7f6520006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:07.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.531+0000 7f65357fa700 1 -- 172.21.15.46:0/1400825589 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f652802d910 con 0x7f6538109800 2024-04-26T21:25:07.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.737+0000 7f653e668700 1 -- 172.21.15.46:0/1400825589 --> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f6538108550 con 0x7f6524039600 2024-04-26T21:25:07.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.977+0000 7f65357fa700 1 -- 172.21.15.46:0/1400825589 <== mgr.14118 v2:172.21.15.46:6800/2053782984 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6538108550 con 0x7f6524039600 2024-04-26T21:25:07.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.979+0000 7f653e668700 1 -- 172.21.15.46:0/1400825589 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f6524039600 msgr2=0x7f652403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:07.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.979+0000 7f653e668700 1 --2- 172.21.15.46:0/1400825589 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f6524039600 0x7f652403bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6520006fd0 tx=0x7f6520006e40 comp rx=0 tx=0).stop 2024-04-26T21:25:07.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.979+0000 7f653e668700 1 -- 172.21.15.46:0/1400825589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6538109800 msgr2=0x7f6538109080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:07.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.979+0000 7f653e668700 1 --2- 172.21.15.46:0/1400825589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6538109800 0x7f6538109080 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6528011ac0 tx=0x7f6528011ba0 comp rx=0 tx=0).stop 2024-04-26T21:25:07.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.979+0000 7f653e668700 1 -- 172.21.15.46:0/1400825589 shutdown_connections 2024-04-26T21:25:07.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.979+0000 7f653e668700 1 --2- 172.21.15.46:0/1400825589 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f6524039600 0x7f652403bac0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:07.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.979+0000 7f653e668700 1 --2- 172.21.15.46:0/1400825589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6538109800 0x7f6538109080 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:07.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.979+0000 7f653e668700 1 -- 172.21.15.46:0/1400825589 >> 172.21.15.46:0/1400825589 conn(0x7f65381031f0 msgr2=0x7f6538103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:07.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.979+0000 7f653e668700 1 -- 172.21.15.46:0/1400825589 shutdown_connections 2024-04-26T21:25:07.708 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:06.979+0000 7f653e668700 1 -- 172.21.15.46:0/1400825589 wait complete. 2024-04-26T21:25:08.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:07 smithi046 ceph-mon[29201]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:25:08.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:07 smithi046 ceph-mon[29201]: Generating ssh key... 2024-04-26T21:25:08.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:07 smithi046 ceph-mon[29201]: from='mgr.14118 172.21.15.46:0/1110106616' entity='mgr.x' 2024-04-26T21:25:08.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:07 smithi046 ceph-mon[29201]: from='mgr.14118 172.21.15.46:0/1110106616' entity='mgr.x' 2024-04-26T21:25:08.970 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+sTE3O9pA8MQ+f+aP4yoPABL5ao5Aejrieq1oKNxiyF5H7Sjxh6lYDvd0gIdnSN73uPMMrJq1tBmlvswAku33BVejwwYDyCrlkRRwQG4IOfPg6mi8/nOtAXBz1Ur2yjtJ7IvB4C7LB5w6gwsMJcklqaBWvHboIeHWo6XAjJwNkrCA2DLIRsGMcDihYMMJiLuu+k/Fed1k9MdAKDxj9Uz7kcIz8DyA7WA2F7lSVaX4YKvo0vS8IcGEinmZ5Hih0E4V/q3JWSJ5ujDwYdKyElxeIeUhMpmrLX8fQ3tXHlJw3IJy/9cDYkrCDuKpUp9GYYQDbLuEGDzaLxrYCJMabgvVdqdH0td9yQvIiTjxcdO00N0LcemrfMTX9OkJvW9CVVl+KEWnJyL3T2bJ4d5Fgo4r8eA4WXgWgBNNwxTl8vIpMpLZCCaVzfIysKXF9k6H5mz8jBZ37JDMl0CNjQThqYe++e1MICbcV/yAx0CtEeSS1Ew9PXpf3FVccU7VOdvZ85s= ceph-36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:25:08.970 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.183+0000 7f99e4f3c700 1 Processor -- start 2024-04-26T21:25:08.970 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.183+0000 7f99e4f3c700 1 -- start start 2024-04-26T21:25:08.970 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.183+0000 7f99e4f3c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99e0108db0 0x7f99e01091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:08.970 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.183+0000 7f99e4f3c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f99e01097a0 con 0x7f99e0108db0 2024-04-26T21:25:08.970 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.183+0000 7f99de59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99e0108db0 0x7f99e01091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:08.971 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.183+0000 7f99de59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99e0108db0 0x7f99e01091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57908/0 (socket says 172.21.15.46:57908) 2024-04-26T21:25:08.971 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.184+0000 7f99de59c700 1 -- 172.21.15.46:0/1490173254 learned_addr learned my addr 172.21.15.46:0/1490173254 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:08.971 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.184+0000 7f99de59c700 1 -- 172.21.15.46:0/1490173254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99e0109f60 con 0x7f99e0108db0 2024-04-26T21:25:08.971 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.184+0000 7f99de59c700 1 --2- 172.21.15.46:0/1490173254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99e0108db0 0x7f99e01091d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f99d00098a0 tx=0x7f99d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=120028d1c8ef96d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:08.971 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.184+0000 7f99dd59a700 1 -- 172.21.15.46:0/1490173254 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99d0004160 con 0x7f99e0108db0 2024-04-26T21:25:08.971 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.184+0000 7f99dd59a700 1 -- 172.21.15.46:0/1490173254 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f99d0004300 con 0x7f99e0108db0 2024-04-26T21:25:08.971 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.184+0000 7f99dd59a700 1 -- 172.21.15.46:0/1490173254 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99d0011af0 con 0x7f99e0108db0 2024-04-26T21:25:08.971 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.185+0000 7f99e4f3c700 1 -- 172.21.15.46:0/1490173254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99e0108db0 msgr2=0x7f99e01091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:08.971 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.185+0000 7f99e4f3c700 1 --2- 172.21.15.46:0/1490173254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99e0108db0 0x7f99e01091d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f99d00098a0 tx=0x7f99d0009bb0 comp rx=0 tx=0).stop 2024-04-26T21:25:08.971 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.185+0000 7f99e4f3c700 1 -- 172.21.15.46:0/1490173254 shutdown_connections 2024-04-26T21:25:08.971 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.185+0000 7f99e4f3c700 1 --2- 172.21.15.46:0/1490173254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99e0108db0 0x7f99e01091d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:08.971 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.185+0000 7f99e4f3c700 1 -- 172.21.15.46:0/1490173254 >> 172.21.15.46:0/1490173254 conn(0x7f99e0104310 msgr2=0x7f99e0106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:08.971 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.185+0000 7f99e4f3c700 1 -- 172.21.15.46:0/1490173254 shutdown_connections 2024-04-26T21:25:08.972 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.185+0000 7f99e4f3c700 1 -- 172.21.15.46:0/1490173254 wait complete. 2024-04-26T21:25:08.972 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.186+0000 7f99e4f3c700 1 Processor -- start 2024-04-26T21:25:08.972 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.186+0000 7f99e4f3c700 1 -- start start 2024-04-26T21:25:08.972 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.186+0000 7f99e4f3c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99e0108db0 0x7f99e007f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:08.972 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.186+0000 7f99e4f3c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f99e010d310 con 0x7f99e0108db0 2024-04-26T21:25:08.972 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.186+0000 7f99de59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99e0108db0 0x7f99e007f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:08.972 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.186+0000 7f99de59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99e0108db0 0x7f99e007f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57914/0 (socket says 172.21.15.46:57914) 2024-04-26T21:25:08.972 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.186+0000 7f99de59c700 1 -- 172.21.15.46:0/437829926 learned_addr learned my addr 172.21.15.46:0/437829926 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:08.972 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.186+0000 7f99de59c700 1 -- 172.21.15.46:0/437829926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99e007f930 con 0x7f99e0108db0 2024-04-26T21:25:08.972 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.186+0000 7f99de59c700 1 --2- 172.21.15.46:0/437829926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99e0108db0 0x7f99e007f3f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f99d00117b0 tx=0x7f99d0011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:08.972 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.186+0000 7f99cf7fe700 1 -- 172.21.15.46:0/437829926 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99d0003af0 con 0x7f99e0108db0 2024-04-26T21:25:08.972 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.187+0000 7f99cf7fe700 1 -- 172.21.15.46:0/437829926 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f99d0020070 con 0x7f99e0108db0 2024-04-26T21:25:08.972 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.187+0000 7f99e4f3c700 1 -- 172.21.15.46:0/437829926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99e007db20 con 0x7f99e0108db0 2024-04-26T21:25:08.972 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.187+0000 7f99cf7fe700 1 -- 172.21.15.46:0/437829926 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99d0019630 con 0x7f99e0108db0 2024-04-26T21:25:08.973 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.187+0000 7f99e4f3c700 1 -- 172.21.15.46:0/437829926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99e007e060 con 0x7f99e0108db0 2024-04-26T21:25:08.973 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.187+0000 7f99cf7fe700 1 -- 172.21.15.46:0/437829926 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f99d00198a0 con 0x7f99e0108db0 2024-04-26T21:25:08.973 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.188+0000 7f99e4f3c700 1 -- 172.21.15.46:0/437829926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99e0064a80 con 0x7f99e0108db0 2024-04-26T21:25:08.973 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.188+0000 7f99cf7fe700 1 --2- 172.21.15.46:0/437829926 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f99c8039600 0x7f99c803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:08.973 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.188+0000 7f99cf7fe700 1 -- 172.21.15.46:0/437829926 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f99d004dce0 con 0x7f99e0108db0 2024-04-26T21:25:08.973 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.188+0000 7f99ddd9b700 1 --2- 172.21.15.46:0/437829926 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f99c8039600 0x7f99c803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:08.973 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.188+0000 7f99ddd9b700 1 --2- 172.21.15.46:0/437829926 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f99c8039600 0x7f99c803bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f99d4006fd0 tx=0x7f99d4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:08.973 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.193+0000 7f99cf7fe700 1 -- 172.21.15.46:0/437829926 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f99d00042a0 con 0x7f99e0108db0 2024-04-26T21:25:08.973 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.401+0000 7f99e4f3c700 1 -- 172.21.15.46:0/437829926 --> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f99e01055f0 con 0x7f99c8039600 2024-04-26T21:25:08.973 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.402+0000 7f99cf7fe700 1 -- 172.21.15.46:0/437829926 <== mgr.14118 v2:172.21.15.46:6800/2053782984 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f99e01055f0 con 0x7f99c8039600 2024-04-26T21:25:08.973 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.404+0000 7f99e4f3c700 1 -- 172.21.15.46:0/437829926 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f99c8039600 msgr2=0x7f99c803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:08.973 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.404+0000 7f99e4f3c700 1 --2- 172.21.15.46:0/437829926 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f99c8039600 0x7f99c803bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f99d4006fd0 tx=0x7f99d4006e40 comp rx=0 tx=0).stop 2024-04-26T21:25:08.973 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.404+0000 7f99e4f3c700 1 -- 172.21.15.46:0/437829926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99e0108db0 msgr2=0x7f99e007f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:08.973 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.404+0000 7f99e4f3c700 1 --2- 172.21.15.46:0/437829926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99e0108db0 0x7f99e007f3f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f99d00117b0 tx=0x7f99d0011b70 comp rx=0 tx=0).stop 2024-04-26T21:25:08.974 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.404+0000 7f99e4f3c700 1 -- 172.21.15.46:0/437829926 shutdown_connections 2024-04-26T21:25:08.974 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.404+0000 7f99e4f3c700 1 --2- 172.21.15.46:0/437829926 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f99c8039600 0x7f99c803bac0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:08.974 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.404+0000 7f99e4f3c700 1 --2- 172.21.15.46:0/437829926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f99e0108db0 0x7f99e007f3f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:08.974 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.404+0000 7f99e4f3c700 1 -- 172.21.15.46:0/437829926 >> 172.21.15.46:0/437829926 conn(0x7f99e0104310 msgr2=0x7f99e0104ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:08.974 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.404+0000 7f99e4f3c700 1 -- 172.21.15.46:0/437829926 shutdown_connections 2024-04-26T21:25:08.974 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:08.404+0000 7f99e4f3c700 1 -- 172.21.15.46:0/437829926 wait complete. 2024-04-26T21:25:08.974 INFO:teuthology.orchestra.run.smithi046.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-26T21:25:08.974 INFO:teuthology.orchestra.run.smithi046.stdout:Adding key to root@localhost authorized_keys... 2024-04-26T21:25:08.974 INFO:teuthology.orchestra.run.smithi046.stdout:Adding host smithi046... 2024-04-26T21:25:09.281 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:09 smithi046 ceph-mon[29201]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:25:10.202 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:10 smithi046 ceph-mon[29201]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi046", "addr": "172.21.15.46", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:25:11.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:11 smithi046 ceph-mon[29201]: Deploying cephadm binary to smithi046 2024-04-26T21:25:14.452 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout Added host 'smithi046' with addr '172.21.15.46' 2024-04-26T21:25:14.453 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.396+0000 7fdd24cf9700 1 Processor -- start 2024-04-26T21:25:14.453 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.396+0000 7fdd24cf9700 1 -- start start 2024-04-26T21:25:14.453 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.396+0000 7fdd24cf9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd20107c50 0x7fdd20108070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:14.453 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.396+0000 7fdd24cf9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdd20108640 con 0x7fdd20107c50 2024-04-26T21:25:14.453 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.396+0000 7fdd1e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd20107c50 0x7fdd20108070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:14.453 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.396+0000 7fdd1e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd20107c50 0x7fdd20108070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57930/0 (socket says 172.21.15.46:57930) 2024-04-26T21:25:14.453 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.396+0000 7fdd1e59c700 1 -- 172.21.15.46:0/455210122 learned_addr learned my addr 172.21.15.46:0/455210122 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:14.453 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.397+0000 7fdd1e59c700 1 -- 172.21.15.46:0/455210122 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd20108e50 con 0x7fdd20107c50 2024-04-26T21:25:14.453 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.397+0000 7fdd1e59c700 1 --2- 172.21.15.46:0/455210122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd20107c50 0x7fdd20108070 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdd100098a0 tx=0x7fdd10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc2a289716f8628f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:14.453 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.397+0000 7fdd1d59a700 1 -- 172.21.15.46:0/455210122 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdd10004160 con 0x7fdd20107c50 2024-04-26T21:25:14.453 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.397+0000 7fdd1d59a700 1 -- 172.21.15.46:0/455210122 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fdd10004300 con 0x7fdd20107c50 2024-04-26T21:25:14.453 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.397+0000 7fdd1d59a700 1 -- 172.21.15.46:0/455210122 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdd10011af0 con 0x7fdd20107c50 2024-04-26T21:25:14.454 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.398+0000 7fdd24cf9700 1 -- 172.21.15.46:0/455210122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd20107c50 msgr2=0x7fdd20108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:14.454 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.398+0000 7fdd24cf9700 1 --2- 172.21.15.46:0/455210122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd20107c50 0x7fdd20108070 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdd100098a0 tx=0x7fdd10009bb0 comp rx=0 tx=0).stop 2024-04-26T21:25:14.454 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.398+0000 7fdd24cf9700 1 -- 172.21.15.46:0/455210122 shutdown_connections 2024-04-26T21:25:14.454 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.398+0000 7fdd24cf9700 1 --2- 172.21.15.46:0/455210122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd20107c50 0x7fdd20108070 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:14.454 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.398+0000 7fdd24cf9700 1 -- 172.21.15.46:0/455210122 >> 172.21.15.46:0/455210122 conn(0x7fdd201031f0 msgr2=0x7fdd20105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:14.454 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.398+0000 7fdd24cf9700 1 -- 172.21.15.46:0/455210122 shutdown_connections 2024-04-26T21:25:14.454 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.398+0000 7fdd24cf9700 1 -- 172.21.15.46:0/455210122 wait complete. 2024-04-26T21:25:14.454 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.399+0000 7fdd24cf9700 1 Processor -- start 2024-04-26T21:25:14.454 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.399+0000 7fdd24cf9700 1 -- start start 2024-04-26T21:25:14.454 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.399+0000 7fdd24cf9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd20107c50 0x7fdd20192dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:14.455 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.399+0000 7fdd24cf9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2010c200 con 0x7fdd20107c50 2024-04-26T21:25:14.455 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.399+0000 7fdd1e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd20107c50 0x7fdd20192dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:14.455 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.399+0000 7fdd1e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd20107c50 0x7fdd20192dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57944/0 (socket says 172.21.15.46:57944) 2024-04-26T21:25:14.455 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.399+0000 7fdd1e59c700 1 -- 172.21.15.46:0/1462984398 learned_addr learned my addr 172.21.15.46:0/1462984398 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:14.455 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.399+0000 7fdd1e59c700 1 -- 172.21.15.46:0/1462984398 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd20193300 con 0x7fdd20107c50 2024-04-26T21:25:14.455 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.399+0000 7fdd1e59c700 1 --2- 172.21.15.46:0/1462984398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd20107c50 0x7fdd20192dc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdd10009fb0 tx=0x7fdd10011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:14.455 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.400+0000 7fdd0f7fe700 1 -- 172.21.15.46:0/1462984398 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdd100036a0 con 0x7fdd20107c50 2024-04-26T21:25:14.455 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.400+0000 7fdd0f7fe700 1 -- 172.21.15.46:0/1462984398 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fdd1001a430 con 0x7fdd20107c50 2024-04-26T21:25:14.455 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.400+0000 7fdd24cf9700 1 -- 172.21.15.46:0/1462984398 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd201914f0 con 0x7fdd20107c50 2024-04-26T21:25:14.455 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.400+0000 7fdd0f7fe700 1 -- 172.21.15.46:0/1462984398 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdd10019550 con 0x7fdd20107c50 2024-04-26T21:25:14.455 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.400+0000 7fdd24cf9700 1 -- 172.21.15.46:0/1462984398 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd20191a30 con 0x7fdd20107c50 2024-04-26T21:25:14.455 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.401+0000 7fdd24cf9700 1 -- 172.21.15.46:0/1462984398 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd20064a80 con 0x7fdd20107c50 2024-04-26T21:25:14.455 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.401+0000 7fdd0f7fe700 1 -- 172.21.15.46:0/1462984398 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fdd10019800 con 0x7fdd20107c50 2024-04-26T21:25:14.455 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.401+0000 7fdd0f7fe700 1 --2- 172.21.15.46:0/1462984398 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fdd080395b0 0x7fdd0803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:14.456 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.401+0000 7fdd0f7fe700 1 -- 172.21.15.46:0/1462984398 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdd1004d9f0 con 0x7fdd20107c50 2024-04-26T21:25:14.456 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.401+0000 7fdd1dd9b700 1 --2- 172.21.15.46:0/1462984398 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fdd080395b0 0x7fdd0803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:14.456 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.401+0000 7fdd1dd9b700 1 --2- 172.21.15.46:0/1462984398 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fdd080395b0 0x7fdd0803ba70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdd14006fd0 tx=0x7fdd14006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:14.456 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.406+0000 7fdd0f7fe700 1 -- 172.21.15.46:0/1462984398 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fdd1001a5e0 con 0x7fdd20107c50 2024-04-26T21:25:14.456 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:09.616+0000 7fdd24cf9700 1 -- 172.21.15.46:0/1462984398 --> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi046", "addr": "172.21.15.46", "target": ["mon-mgr", ""]}) -- 0x7fdd20196bd0 con 0x7fdd080395b0 2024-04-26T21:25:14.456 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:13.847+0000 7fdd0f7fe700 1 -- 172.21.15.46:0/1462984398 <== mgr.14118 v2:172.21.15.46:6800/2053782984 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fdd20196bd0 con 0x7fdd080395b0 2024-04-26T21:25:14.456 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:13.850+0000 7fdd24cf9700 1 -- 172.21.15.46:0/1462984398 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fdd080395b0 msgr2=0x7fdd0803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:14.456 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:13.850+0000 7fdd24cf9700 1 --2- 172.21.15.46:0/1462984398 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fdd080395b0 0x7fdd0803ba70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdd14006fd0 tx=0x7fdd14006e40 comp rx=0 tx=0).stop 2024-04-26T21:25:14.456 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:13.850+0000 7fdd24cf9700 1 -- 172.21.15.46:0/1462984398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd20107c50 msgr2=0x7fdd20192dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:14.456 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:13.850+0000 7fdd24cf9700 1 --2- 172.21.15.46:0/1462984398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd20107c50 0x7fdd20192dc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdd10009fb0 tx=0x7fdd10011940 comp rx=0 tx=0).stop 2024-04-26T21:25:14.456 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:13.850+0000 7fdd24cf9700 1 -- 172.21.15.46:0/1462984398 shutdown_connections 2024-04-26T21:25:14.456 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:13.850+0000 7fdd24cf9700 1 --2- 172.21.15.46:0/1462984398 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fdd080395b0 0x7fdd0803ba70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:14.456 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:13.850+0000 7fdd24cf9700 1 --2- 172.21.15.46:0/1462984398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd20107c50 0x7fdd20192dc0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:14.456 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:13.850+0000 7fdd24cf9700 1 -- 172.21.15.46:0/1462984398 >> 172.21.15.46:0/1462984398 conn(0x7fdd201031f0 msgr2=0x7fdd20104100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:14.456 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:13.850+0000 7fdd24cf9700 1 -- 172.21.15.46:0/1462984398 shutdown_connections 2024-04-26T21:25:14.457 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:13.850+0000 7fdd24cf9700 1 -- 172.21.15.46:0/1462984398 wait complete. 2024-04-26T21:25:14.457 INFO:teuthology.orchestra.run.smithi046.stdout:Deploying unmanaged mon service... 2024-04-26T21:25:15.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:14 smithi046 ceph-mon[29201]: from='mgr.14118 172.21.15.46:0/1110106616' entity='mgr.x' 2024-04-26T21:25:15.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:14 smithi046 ceph-mon[29201]: Added host smithi046 2024-04-26T21:25:15.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:14 smithi046 ceph-mon[29201]: from='mgr.14118 172.21.15.46:0/1110106616' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:16.492 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:16 smithi046 ceph-mon[29201]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:25:16.492 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:16 smithi046 ceph-mon[29201]: Saving service mon spec with placement count:5 2024-04-26T21:25:16.492 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:16 smithi046 ceph-mon[29201]: from='mgr.14118 172.21.15.46:0/1110106616' entity='mgr.x' 2024-04-26T21:25:16.809 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-26T21:25:16.809 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.869+0000 7ffb4bdd0700 1 Processor -- start 2024-04-26T21:25:16.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.869+0000 7ffb4bdd0700 1 -- start start 2024-04-26T21:25:16.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.869+0000 7ffb4bdd0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb44107690 0x7ffb44107ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:16.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.869+0000 7ffb4bdd0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ffb4407ce00 con 0x7ffb44107690 2024-04-26T21:25:16.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.870+0000 7ffb49b6c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb44107690 0x7ffb44107ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:16.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.870+0000 7ffb49b6c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb44107690 0x7ffb44107ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57960/0 (socket says 172.21.15.46:57960) 2024-04-26T21:25:16.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.870+0000 7ffb49b6c700 1 -- 172.21.15.46:0/1680103436 learned_addr learned my addr 172.21.15.46:0/1680103436 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:16.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.870+0000 7ffb49b6c700 1 -- 172.21.15.46:0/1680103436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb44107ff0 con 0x7ffb44107690 2024-04-26T21:25:16.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.870+0000 7ffb49b6c700 1 --2- 172.21.15.46:0/1680103436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb44107690 0x7ffb44107ab0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ffb34009e80 tx=0x7ffb3400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59ad1ec802cca065 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:16.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.870+0000 7ffb48b6a700 1 -- 172.21.15.46:0/1680103436 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb34017070 con 0x7ffb44107690 2024-04-26T21:25:16.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.870+0000 7ffb48b6a700 1 -- 172.21.15.46:0/1680103436 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ffb3400b8c0 con 0x7ffb44107690 2024-04-26T21:25:16.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.870+0000 7ffb48b6a700 1 -- 172.21.15.46:0/1680103436 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb34004490 con 0x7ffb44107690 2024-04-26T21:25:16.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.871+0000 7ffb4bdd0700 1 -- 172.21.15.46:0/1680103436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb44107690 msgr2=0x7ffb44107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:16.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.871+0000 7ffb4bdd0700 1 --2- 172.21.15.46:0/1680103436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb44107690 0x7ffb44107ab0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ffb34009e80 tx=0x7ffb3400b330 comp rx=0 tx=0).stop 2024-04-26T21:25:16.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.871+0000 7ffb4bdd0700 1 -- 172.21.15.46:0/1680103436 shutdown_connections 2024-04-26T21:25:16.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.871+0000 7ffb4bdd0700 1 --2- 172.21.15.46:0/1680103436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb44107690 0x7ffb44107ab0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:16.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.871+0000 7ffb4bdd0700 1 -- 172.21.15.46:0/1680103436 >> 172.21.15.46:0/1680103436 conn(0x7ffb441031f0 msgr2=0x7ffb44105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:16.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.871+0000 7ffb4bdd0700 1 -- 172.21.15.46:0/1680103436 shutdown_connections 2024-04-26T21:25:16.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.871+0000 7ffb4bdd0700 1 -- 172.21.15.46:0/1680103436 wait complete. 2024-04-26T21:25:16.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.872+0000 7ffb4bdd0700 1 Processor -- start 2024-04-26T21:25:16.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.872+0000 7ffb4bdd0700 1 -- start start 2024-04-26T21:25:16.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.872+0000 7ffb4bdd0700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb44107690 0x7ffb4419eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:16.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.872+0000 7ffb4bdd0700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ffb44109020 con 0x7ffb44107690 2024-04-26T21:25:16.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.872+0000 7ffb49b6c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb44107690 0x7ffb4419eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:16.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.872+0000 7ffb49b6c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb44107690 0x7ffb4419eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57964/0 (socket says 172.21.15.46:57964) 2024-04-26T21:25:16.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.872+0000 7ffb49b6c700 1 -- 172.21.15.46:0/1172806233 learned_addr learned my addr 172.21.15.46:0/1172806233 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:16.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.872+0000 7ffb49b6c700 1 -- 172.21.15.46:0/1172806233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb4419f3f0 con 0x7ffb44107690 2024-04-26T21:25:16.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.873+0000 7ffb49b6c700 1 --2- 172.21.15.46:0/1172806233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb44107690 0x7ffb4419eeb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ffb3400b7b0 tx=0x7ffb3400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:16.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.873+0000 7ffb42ffd700 1 -- 172.21.15.46:0/1172806233 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb34020020 con 0x7ffb44107690 2024-04-26T21:25:16.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.873+0000 7ffb42ffd700 1 -- 172.21.15.46:0/1172806233 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ffb34012430 con 0x7ffb44107690 2024-04-26T21:25:16.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.873+0000 7ffb4bdd0700 1 -- 172.21.15.46:0/1172806233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb4419f680 con 0x7ffb44107690 2024-04-26T21:25:16.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.873+0000 7ffb42ffd700 1 -- 172.21.15.46:0/1172806233 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb34017040 con 0x7ffb44107690 2024-04-26T21:25:16.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.873+0000 7ffb4bdd0700 1 -- 172.21.15.46:0/1172806233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb441a2370 con 0x7ffb44107690 2024-04-26T21:25:16.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.874+0000 7ffb42ffd700 1 -- 172.21.15.46:0/1172806233 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7ffb34023020 con 0x7ffb44107690 2024-04-26T21:25:16.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.874+0000 7ffb4bdd0700 1 -- 172.21.15.46:0/1172806233 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb44064a80 con 0x7ffb44107690 2024-04-26T21:25:16.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.874+0000 7ffb42ffd700 1 --2- 172.21.15.46:0/1172806233 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7ffb2c039560 0x7ffb2c03ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:16.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.874+0000 7ffb42ffd700 1 -- 172.21.15.46:0/1172806233 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ffb3400f050 con 0x7ffb44107690 2024-04-26T21:25:16.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.874+0000 7ffb4936b700 1 --2- 172.21.15.46:0/1172806233 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7ffb2c039560 0x7ffb2c03ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:16.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.874+0000 7ffb4936b700 1 --2- 172.21.15.46:0/1172806233 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7ffb2c039560 0x7ffb2c03ba20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ffb38006fd0 tx=0x7ffb38006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:16.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:14.879+0000 7ffb42ffd700 1 -- 172.21.15.46:0/1172806233 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ffb3402a020 con 0x7ffb44107690 2024-04-26T21:25:16.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:15.106+0000 7ffb4bdd0700 1 -- 172.21.15.46:0/1172806233 --> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7ffb4402e870 con 0x7ffb2c039560 2024-04-26T21:25:16.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:15.111+0000 7ffb42ffd700 1 -- 172.21.15.46:0/1172806233 <== mgr.14118 v2:172.21.15.46:6800/2053782984 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7ffb4402e870 con 0x7ffb2c039560 2024-04-26T21:25:16.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:15.113+0000 7ffb4bdd0700 1 -- 172.21.15.46:0/1172806233 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7ffb2c039560 msgr2=0x7ffb2c03ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:16.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:15.113+0000 7ffb4bdd0700 1 --2- 172.21.15.46:0/1172806233 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7ffb2c039560 0x7ffb2c03ba20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ffb38006fd0 tx=0x7ffb38006e40 comp rx=0 tx=0).stop 2024-04-26T21:25:16.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:15.113+0000 7ffb4bdd0700 1 -- 172.21.15.46:0/1172806233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb44107690 msgr2=0x7ffb4419eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:16.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:15.113+0000 7ffb4bdd0700 1 --2- 172.21.15.46:0/1172806233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb44107690 0x7ffb4419eeb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ffb3400b7b0 tx=0x7ffb3400be20 comp rx=0 tx=0).stop 2024-04-26T21:25:16.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:15.113+0000 7ffb4bdd0700 1 -- 172.21.15.46:0/1172806233 shutdown_connections 2024-04-26T21:25:16.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:15.113+0000 7ffb4bdd0700 1 --2- 172.21.15.46:0/1172806233 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7ffb2c039560 0x7ffb2c03ba20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:16.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:15.113+0000 7ffb4bdd0700 1 --2- 172.21.15.46:0/1172806233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb44107690 0x7ffb4419eeb0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:16.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:15.113+0000 7ffb4bdd0700 1 -- 172.21.15.46:0/1172806233 >> 172.21.15.46:0/1172806233 conn(0x7ffb441031f0 msgr2=0x7ffb44195ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:16.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:15.113+0000 7ffb4bdd0700 1 -- 172.21.15.46:0/1172806233 shutdown_connections 2024-04-26T21:25:16.814 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:15.113+0000 7ffb4bdd0700 1 -- 172.21.15.46:0/1172806233 wait complete. 2024-04-26T21:25:16.814 INFO:teuthology.orchestra.run.smithi046.stdout:Deploying unmanaged mgr service... 2024-04-26T21:25:17.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:17 smithi046 ceph-mon[29201]: from='mgr.14118 172.21.15.46:0/1110106616' entity='mgr.x' 2024-04-26T21:25:17.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:17 smithi046 ceph-mon[29201]: from='mgr.14118 172.21.15.46:0/1110106616' entity='mgr.x' 2024-04-26T21:25:18.737 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:18 smithi046 ceph-mon[29201]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:25:18.738 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:18 smithi046 ceph-mon[29201]: Saving service mgr spec with placement count:2 2024-04-26T21:25:18.738 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:18 smithi046 ceph-mon[29201]: from='mgr.14118 172.21.15.46:0/1110106616' entity='mgr.x' 2024-04-26T21:25:18.745 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-26T21:25:18.745 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.266+0000 7feda8e6f700 1 Processor -- start 2024-04-26T21:25:18.745 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.266+0000 7feda8e6f700 1 -- start start 2024-04-26T21:25:18.745 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.266+0000 7feda8e6f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feda4107610 0x7feda4107a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:18.745 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.266+0000 7feda8e6f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7feda407d6d0 con 0x7feda4107610 2024-04-26T21:25:18.746 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.266+0000 7feda259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feda4107610 0x7feda4107a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:18.746 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.266+0000 7feda259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feda4107610 0x7feda4107a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37404/0 (socket says 172.21.15.46:37404) 2024-04-26T21:25:18.746 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.266+0000 7feda259c700 1 -- 172.21.15.46:0/401229870 learned_addr learned my addr 172.21.15.46:0/401229870 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:18.746 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.267+0000 7feda259c700 1 -- 172.21.15.46:0/401229870 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feda4107f70 con 0x7feda4107610 2024-04-26T21:25:18.746 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.267+0000 7feda259c700 1 --2- 172.21.15.46:0/401229870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feda4107610 0x7feda4107a30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fed940098a0 tx=0x7fed94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b0d6287ab4e498 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:18.746 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.267+0000 7feda159a700 1 -- 172.21.15.46:0/401229870 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed94004160 con 0x7feda4107610 2024-04-26T21:25:18.746 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.267+0000 7feda159a700 1 -- 172.21.15.46:0/401229870 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed94004300 con 0x7feda4107610 2024-04-26T21:25:18.746 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.267+0000 7feda159a700 1 -- 172.21.15.46:0/401229870 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed94011a20 con 0x7feda4107610 2024-04-26T21:25:18.746 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.268+0000 7feda8e6f700 1 -- 172.21.15.46:0/401229870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feda4107610 msgr2=0x7feda4107a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:18.746 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.268+0000 7feda8e6f700 1 --2- 172.21.15.46:0/401229870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feda4107610 0x7feda4107a30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fed940098a0 tx=0x7fed94009bb0 comp rx=0 tx=0).stop 2024-04-26T21:25:18.746 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.268+0000 7feda8e6f700 1 -- 172.21.15.46:0/401229870 shutdown_connections 2024-04-26T21:25:18.746 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.268+0000 7feda8e6f700 1 --2- 172.21.15.46:0/401229870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feda4107610 0x7feda4107a30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:18.746 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.268+0000 7feda8e6f700 1 -- 172.21.15.46:0/401229870 >> 172.21.15.46:0/401229870 conn(0x7feda41031f0 msgr2=0x7feda4105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:18.746 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.268+0000 7feda8e6f700 1 -- 172.21.15.46:0/401229870 shutdown_connections 2024-04-26T21:25:18.747 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.268+0000 7feda8e6f700 1 -- 172.21.15.46:0/401229870 wait complete. 2024-04-26T21:25:18.747 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.269+0000 7feda8e6f700 1 Processor -- start 2024-04-26T21:25:18.747 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.269+0000 7feda8e6f700 1 -- start start 2024-04-26T21:25:18.747 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.269+0000 7feda8e6f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feda4107610 0x7feda419ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:18.747 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.269+0000 7feda8e6f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7feda4108fe0 con 0x7feda4107610 2024-04-26T21:25:18.747 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.269+0000 7feda259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feda4107610 0x7feda419ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:18.747 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.269+0000 7feda259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feda4107610 0x7feda419ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37414/0 (socket says 172.21.15.46:37414) 2024-04-26T21:25:18.747 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.269+0000 7feda259c700 1 -- 172.21.15.46:0/2947962717 learned_addr learned my addr 172.21.15.46:0/2947962717 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:18.747 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.270+0000 7feda259c700 1 -- 172.21.15.46:0/2947962717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feda419f380 con 0x7feda4107610 2024-04-26T21:25:18.747 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.270+0000 7feda259c700 1 --2- 172.21.15.46:0/2947962717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feda4107610 0x7feda419ee40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fed94004660 tx=0x7fed94011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:18.747 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.270+0000 7fed937fe700 1 -- 172.21.15.46:0/2947962717 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed94003aa0 con 0x7feda4107610 2024-04-26T21:25:18.747 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.270+0000 7feda8e6f700 1 -- 172.21.15.46:0/2947962717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feda419f610 con 0x7feda4107610 2024-04-26T21:25:18.747 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.270+0000 7fed937fe700 1 -- 172.21.15.46:0/2947962717 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed9401b8f0 con 0x7feda4107610 2024-04-26T21:25:18.747 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.270+0000 7fed937fe700 1 -- 172.21.15.46:0/2947962717 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed94023900 con 0x7feda4107610 2024-04-26T21:25:18.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.270+0000 7feda8e6f700 1 -- 172.21.15.46:0/2947962717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feda41a2300 con 0x7feda4107610 2024-04-26T21:25:18.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.271+0000 7fed937fe700 1 -- 172.21.15.46:0/2947962717 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fed94023b70 con 0x7feda4107610 2024-04-26T21:25:18.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.271+0000 7fed937fe700 1 --2- 172.21.15.46:0/2947962717 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fed8c039600 0x7fed8c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:18.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.271+0000 7fed937fe700 1 -- 172.21.15.46:0/2947962717 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fed9404df40 con 0x7feda4107610 2024-04-26T21:25:18.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.271+0000 7feda1d9b700 1 --2- 172.21.15.46:0/2947962717 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fed8c039600 0x7fed8c03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:18.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.271+0000 7feda8e6f700 1 -- 172.21.15.46:0/2947962717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feda4064a80 con 0x7feda4107610 2024-04-26T21:25:18.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.272+0000 7feda1d9b700 1 --2- 172.21.15.46:0/2947962717 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fed8c039600 0x7fed8c03bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fed98006fd0 tx=0x7fed98006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:18.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.277+0000 7fed937fe700 1 -- 172.21.15.46:0/2947962717 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fed9401f030 con 0x7feda4107610 2024-04-26T21:25:18.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.522+0000 7feda8e6f700 1 -- 172.21.15.46:0/2947962717 --> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7feda402e750 con 0x7fed8c039600 2024-04-26T21:25:18.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.527+0000 7fed937fe700 1 -- 172.21.15.46:0/2947962717 <== mgr.14118 v2:172.21.15.46:6800/2053782984 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7feda402e750 con 0x7fed8c039600 2024-04-26T21:25:18.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.529+0000 7feda8e6f700 1 -- 172.21.15.46:0/2947962717 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fed8c039600 msgr2=0x7fed8c03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:18.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.529+0000 7feda8e6f700 1 --2- 172.21.15.46:0/2947962717 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fed8c039600 0x7fed8c03bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fed98006fd0 tx=0x7fed98006e40 comp rx=0 tx=0).stop 2024-04-26T21:25:18.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.529+0000 7feda8e6f700 1 -- 172.21.15.46:0/2947962717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feda4107610 msgr2=0x7feda419ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:18.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.529+0000 7feda8e6f700 1 --2- 172.21.15.46:0/2947962717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feda4107610 0x7feda419ee40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fed94004660 tx=0x7fed94011aa0 comp rx=0 tx=0).stop 2024-04-26T21:25:18.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.529+0000 7feda8e6f700 1 -- 172.21.15.46:0/2947962717 shutdown_connections 2024-04-26T21:25:18.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.529+0000 7feda8e6f700 1 --2- 172.21.15.46:0/2947962717 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fed8c039600 0x7fed8c03bac0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:18.749 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.529+0000 7feda8e6f700 1 --2- 172.21.15.46:0/2947962717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feda4107610 0x7feda419ee40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:18.749 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.529+0000 7feda8e6f700 1 -- 172.21.15.46:0/2947962717 >> 172.21.15.46:0/2947962717 conn(0x7feda41031f0 msgr2=0x7feda4195b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:18.749 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.529+0000 7feda8e6f700 1 -- 172.21.15.46:0/2947962717 shutdown_connections 2024-04-26T21:25:18.749 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:17.529+0000 7feda8e6f700 1 -- 172.21.15.46:0/2947962717 wait complete. 2024-04-26T21:25:19.991 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.167+0000 7f054e7e8700 1 Processor -- start 2024-04-26T21:25:19.991 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.167+0000 7f054e7e8700 1 -- start start 2024-04-26T21:25:19.991 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.168+0000 7f054e7e8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0548107070 0x7f0548107490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:19.991 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.168+0000 7f054e7e8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0548107a60 con 0x7f0548107070 2024-04-26T21:25:19.992 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.168+0000 7f0547fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0548107070 0x7f0548107490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:19.992 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.168+0000 7f0547fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0548107070 0x7f0548107490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37422/0 (socket says 172.21.15.46:37422) 2024-04-26T21:25:19.992 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.168+0000 7f0547fff700 1 -- 172.21.15.46:0/3369508510 learned_addr learned my addr 172.21.15.46:0/3369508510 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:19.992 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.168+0000 7f0547fff700 1 -- 172.21.15.46:0/3369508510 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0548107be0 con 0x7f0548107070 2024-04-26T21:25:19.992 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.168+0000 7f0547fff700 1 --2- 172.21.15.46:0/3369508510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0548107070 0x7f0548107490 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f05380098a0 tx=0x7f0538009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dacc6eb2dbc109aa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:19.992 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.168+0000 7f0546ffd700 1 -- 172.21.15.46:0/3369508510 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0538004160 con 0x7f0548107070 2024-04-26T21:25:19.992 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.168+0000 7f0546ffd700 1 -- 172.21.15.46:0/3369508510 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0538004300 con 0x7f0548107070 2024-04-26T21:25:19.992 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.169+0000 7f0546ffd700 1 -- 172.21.15.46:0/3369508510 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0538011a20 con 0x7f0548107070 2024-04-26T21:25:19.992 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.169+0000 7f054e7e8700 1 -- 172.21.15.46:0/3369508510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0548107070 msgr2=0x7f0548107490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:19.992 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.169+0000 7f054e7e8700 1 --2- 172.21.15.46:0/3369508510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0548107070 0x7f0548107490 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f05380098a0 tx=0x7f0538009bb0 comp rx=0 tx=0).stop 2024-04-26T21:25:19.992 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.169+0000 7f054e7e8700 1 -- 172.21.15.46:0/3369508510 shutdown_connections 2024-04-26T21:25:19.992 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.169+0000 7f054e7e8700 1 --2- 172.21.15.46:0/3369508510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0548107070 0x7f0548107490 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:19.992 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.169+0000 7f054e7e8700 1 -- 172.21.15.46:0/3369508510 >> 172.21.15.46:0/3369508510 conn(0x7f0548102970 msgr2=0x7f0548104df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:19.992 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.169+0000 7f054e7e8700 1 -- 172.21.15.46:0/3369508510 shutdown_connections 2024-04-26T21:25:19.993 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.169+0000 7f054e7e8700 1 -- 172.21.15.46:0/3369508510 wait complete. 2024-04-26T21:25:19.993 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.170+0000 7f054e7e8700 1 Processor -- start 2024-04-26T21:25:19.993 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.170+0000 7f054e7e8700 1 -- start start 2024-04-26T21:25:19.993 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.170+0000 7f054e7e8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0548107070 0x7f0548109970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:19.993 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.170+0000 7f054e7e8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f054810af80 con 0x7f0548107070 2024-04-26T21:25:19.993 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.170+0000 7f0547fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0548107070 0x7f0548109970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:19.993 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.170+0000 7f0547fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0548107070 0x7f0548109970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37434/0 (socket says 172.21.15.46:37434) 2024-04-26T21:25:19.993 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.170+0000 7f0547fff700 1 -- 172.21.15.46:0/55400355 learned_addr learned my addr 172.21.15.46:0/55400355 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:19.993 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.171+0000 7f0547fff700 1 -- 172.21.15.46:0/55400355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0548109eb0 con 0x7f0548107070 2024-04-26T21:25:19.993 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.171+0000 7f0547fff700 1 --2- 172.21.15.46:0/55400355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0548107070 0x7f0548109970 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f05380119f0 tx=0x7f0538011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:19.993 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.171+0000 7f05457fa700 1 -- 172.21.15.46:0/55400355 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f05380039e0 con 0x7f0548107070 2024-04-26T21:25:19.993 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.171+0000 7f05457fa700 1 -- 172.21.15.46:0/55400355 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f053801b8f0 con 0x7f0548107070 2024-04-26T21:25:19.993 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.171+0000 7f054e7e8700 1 -- 172.21.15.46:0/55400355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05481080a0 con 0x7f0548107070 2024-04-26T21:25:19.993 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.171+0000 7f05457fa700 1 -- 172.21.15.46:0/55400355 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f05380237a0 con 0x7f0548107070 2024-04-26T21:25:19.993 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.171+0000 7f054e7e8700 1 -- 172.21.15.46:0/55400355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05481085e0 con 0x7f0548107070 2024-04-26T21:25:19.994 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.172+0000 7f05457fa700 1 -- 172.21.15.46:0/55400355 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f053802c3f0 con 0x7f0548107070 2024-04-26T21:25:19.994 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.172+0000 7f05457fa700 1 --2- 172.21.15.46:0/55400355 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f0530039560 0x7f053003ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:19.994 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.172+0000 7f05457fa700 1 -- 172.21.15.46:0/55400355 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f053804de70 con 0x7f0548107070 2024-04-26T21:25:19.994 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.172+0000 7f054e7e8700 1 -- 172.21.15.46:0/55400355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0548064a80 con 0x7f0548107070 2024-04-26T21:25:19.994 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.172+0000 7f05477fe700 1 --2- 172.21.15.46:0/55400355 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f0530039560 0x7f053003ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:19.994 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.173+0000 7f05477fe700 1 --2- 172.21.15.46:0/55400355 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f0530039560 0x7f053003ba20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f053c006fd0 tx=0x7f053c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:19.994 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.177+0000 7f05457fa700 1 -- 172.21.15.46:0/55400355 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f053801f030 con 0x7f0548107070 2024-04-26T21:25:19.994 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.388+0000 7f054e7e8700 1 -- 172.21.15.46:0/55400355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f05481088c0 con 0x7f0548107070 2024-04-26T21:25:19.994 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.390+0000 7f05457fa700 1 -- 172.21.15.46:0/55400355 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f05380202b0 con 0x7f0548107070 2024-04-26T21:25:19.994 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.393+0000 7f054e7e8700 1 -- 172.21.15.46:0/55400355 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f0530039560 msgr2=0x7f053003ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:19.994 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.393+0000 7f054e7e8700 1 --2- 172.21.15.46:0/55400355 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f0530039560 0x7f053003ba20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f053c006fd0 tx=0x7f053c006e40 comp rx=0 tx=0).stop 2024-04-26T21:25:19.994 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.393+0000 7f054e7e8700 1 -- 172.21.15.46:0/55400355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0548107070 msgr2=0x7f0548109970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:19.994 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.393+0000 7f054e7e8700 1 --2- 172.21.15.46:0/55400355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0548107070 0x7f0548109970 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f05380119f0 tx=0x7f0538011ad0 comp rx=0 tx=0).stop 2024-04-26T21:25:19.994 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.393+0000 7f054e7e8700 1 -- 172.21.15.46:0/55400355 shutdown_connections 2024-04-26T21:25:19.995 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.393+0000 7f054e7e8700 1 --2- 172.21.15.46:0/55400355 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f0530039560 0x7f053003ba20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:19.995 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.393+0000 7f054e7e8700 1 --2- 172.21.15.46:0/55400355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0548107070 0x7f0548109970 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:19.995 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.393+0000 7f054e7e8700 1 -- 172.21.15.46:0/55400355 >> 172.21.15.46:0/55400355 conn(0x7f0548102970 msgr2=0x7f05481034a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:19.995 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.393+0000 7f054e7e8700 1 -- 172.21.15.46:0/55400355 shutdown_connections 2024-04-26T21:25:19.995 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:19.393+0000 7f054e7e8700 1 -- 172.21.15.46:0/55400355 wait complete. 2024-04-26T21:25:20.538 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:20 smithi046 ceph-mon[29201]: from='mgr.14118 172.21.15.46:0/1110106616' entity='mgr.x' 2024-04-26T21:25:20.538 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:20 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/55400355' entity='client.admin' 2024-04-26T21:25:21.330 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.526+0000 7f5eaa828700 1 Processor -- start 2024-04-26T21:25:21.330 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.526+0000 7f5eaa828700 1 -- start start 2024-04-26T21:25:21.331 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.526+0000 7f5eaa828700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ea4109860 0x7f5ea410bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:21.331 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.527+0000 7f5eaa828700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea407ce40 con 0x7f5ea4109860 2024-04-26T21:25:21.331 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.527+0000 7f5ea3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ea4109860 0x7f5ea410bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:21.331 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.527+0000 7f5ea3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ea4109860 0x7f5ea410bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37436/0 (socket says 172.21.15.46:37436) 2024-04-26T21:25:21.331 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.527+0000 7f5ea3fff700 1 -- 172.21.15.46:0/1912088131 learned_addr learned my addr 172.21.15.46:0/1912088131 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:21.331 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.527+0000 7f5ea3fff700 1 -- 172.21.15.46:0/1912088131 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ea410c190 con 0x7f5ea4109860 2024-04-26T21:25:21.331 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.527+0000 7f5ea3fff700 1 --2- 172.21.15.46:0/1912088131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ea4109860 0x7f5ea410bc50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c0098a0 tx=0x7f5e8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=324a7fa2a8dc124 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:21.331 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.527+0000 7f5ea2ffd700 1 -- 172.21.15.46:0/1912088131 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e8c004160 con 0x7f5ea4109860 2024-04-26T21:25:21.331 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.527+0000 7f5ea2ffd700 1 -- 172.21.15.46:0/1912088131 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e8c004300 con 0x7f5ea4109860 2024-04-26T21:25:21.331 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.528+0000 7f5ea2ffd700 1 -- 172.21.15.46:0/1912088131 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e8c011a20 con 0x7f5ea4109860 2024-04-26T21:25:21.331 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.528+0000 7f5eaa828700 1 -- 172.21.15.46:0/1912088131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ea4109860 msgr2=0x7f5ea410bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:21.331 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.528+0000 7f5eaa828700 1 --2- 172.21.15.46:0/1912088131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ea4109860 0x7f5ea410bc50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c0098a0 tx=0x7f5e8c009bb0 comp rx=0 tx=0).stop 2024-04-26T21:25:21.331 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.528+0000 7f5eaa828700 1 -- 172.21.15.46:0/1912088131 shutdown_connections 2024-04-26T21:25:21.331 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.528+0000 7f5eaa828700 1 --2- 172.21.15.46:0/1912088131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ea4109860 0x7f5ea410bc50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:21.331 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.528+0000 7f5eaa828700 1 -- 172.21.15.46:0/1912088131 >> 172.21.15.46:0/1912088131 conn(0x7f5ea41031f0 msgr2=0x7f5ea4105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:21.332 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.528+0000 7f5eaa828700 1 -- 172.21.15.46:0/1912088131 shutdown_connections 2024-04-26T21:25:21.332 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.528+0000 7f5eaa828700 1 -- 172.21.15.46:0/1912088131 wait complete. 2024-04-26T21:25:21.332 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.529+0000 7f5eaa828700 1 Processor -- start 2024-04-26T21:25:21.332 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.529+0000 7f5eaa828700 1 -- start start 2024-04-26T21:25:21.332 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.529+0000 7f5eaa828700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ea4109860 0x7f5ea419cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:21.332 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.529+0000 7f5eaa828700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea410d140 con 0x7f5ea4109860 2024-04-26T21:25:21.332 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.530+0000 7f5ea3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ea4109860 0x7f5ea419cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:21.332 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.530+0000 7f5ea3fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ea4109860 0x7f5ea419cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37452/0 (socket says 172.21.15.46:37452) 2024-04-26T21:25:21.332 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.530+0000 7f5ea3fff700 1 -- 172.21.15.46:0/4199348586 learned_addr learned my addr 172.21.15.46:0/4199348586 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:21.332 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.530+0000 7f5ea3fff700 1 -- 172.21.15.46:0/4199348586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ea419d1e0 con 0x7f5ea4109860 2024-04-26T21:25:21.332 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.530+0000 7f5ea3fff700 1 --2- 172.21.15.46:0/4199348586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ea4109860 0x7f5ea419cca0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c004660 tx=0x7f5e8c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:21.332 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.530+0000 7f5ea17fa700 1 -- 172.21.15.46:0/4199348586 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e8c003aa0 con 0x7f5ea4109860 2024-04-26T21:25:21.332 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.530+0000 7f5ea17fa700 1 -- 172.21.15.46:0/4199348586 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e8c01b8f0 con 0x7f5ea4109860 2024-04-26T21:25:21.332 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.530+0000 7f5eaa828700 1 -- 172.21.15.46:0/4199348586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ea419d470 con 0x7f5ea4109860 2024-04-26T21:25:21.332 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.530+0000 7f5ea17fa700 1 -- 172.21.15.46:0/4199348586 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e8c023900 con 0x7f5ea4109860 2024-04-26T21:25:21.332 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.530+0000 7f5eaa828700 1 -- 172.21.15.46:0/4199348586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ea41a0160 con 0x7f5ea4109860 2024-04-26T21:25:21.333 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.531+0000 7f5ea17fa700 1 -- 172.21.15.46:0/4199348586 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f5e8c023bc0 con 0x7f5ea4109860 2024-04-26T21:25:21.333 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.531+0000 7f5ea17fa700 1 --2- 172.21.15.46:0/4199348586 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f5e900395b0 0x7f5e9003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:21.333 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.531+0000 7f5ea17fa700 1 -- 172.21.15.46:0/4199348586 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5e8c04eea0 con 0x7f5ea4109860 2024-04-26T21:25:21.333 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.532+0000 7f5ea37fe700 1 --2- 172.21.15.46:0/4199348586 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f5e900395b0 0x7f5e9003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:21.333 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.532+0000 7f5eaa828700 1 -- 172.21.15.46:0/4199348586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ea4064a80 con 0x7f5ea4109860 2024-04-26T21:25:21.333 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.532+0000 7f5ea37fe700 1 --2- 172.21.15.46:0/4199348586 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f5e900395b0 0x7f5e9003ba70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5e94006fd0 tx=0x7f5e94006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:21.333 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.537+0000 7f5ea17fa700 1 -- 172.21.15.46:0/4199348586 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5e8c01f030 con 0x7f5ea4109860 2024-04-26T21:25:21.333 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.749+0000 7f5eaa828700 1 -- 172.21.15.46:0/4199348586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f5ea41a0330 con 0x7f5ea4109860 2024-04-26T21:25:21.333 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.751+0000 7f5ea17fa700 1 -- 172.21.15.46:0/4199348586 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f5e8c010380 con 0x7f5ea4109860 2024-04-26T21:25:21.333 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.754+0000 7f5eaa828700 1 -- 172.21.15.46:0/4199348586 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f5e900395b0 msgr2=0x7f5e9003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:21.333 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.754+0000 7f5eaa828700 1 --2- 172.21.15.46:0/4199348586 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f5e900395b0 0x7f5e9003ba70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5e94006fd0 tx=0x7f5e94006e40 comp rx=0 tx=0).stop 2024-04-26T21:25:21.333 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.754+0000 7f5eaa828700 1 -- 172.21.15.46:0/4199348586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ea4109860 msgr2=0x7f5ea419cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:21.333 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.754+0000 7f5eaa828700 1 --2- 172.21.15.46:0/4199348586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ea4109860 0x7f5ea419cca0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c004660 tx=0x7f5e8c011aa0 comp rx=0 tx=0).stop 2024-04-26T21:25:21.333 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.754+0000 7f5eaa828700 1 -- 172.21.15.46:0/4199348586 shutdown_connections 2024-04-26T21:25:21.333 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.754+0000 7f5eaa828700 1 --2- 172.21.15.46:0/4199348586 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f5e900395b0 0x7f5e9003ba70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:21.334 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.754+0000 7f5eaa828700 1 --2- 172.21.15.46:0/4199348586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ea4109860 0x7f5ea419cca0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:21.334 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.754+0000 7f5eaa828700 1 -- 172.21.15.46:0/4199348586 >> 172.21.15.46:0/4199348586 conn(0x7f5ea41031f0 msgr2=0x7f5ea4103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:21.334 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.754+0000 7f5eaa828700 1 -- 172.21.15.46:0/4199348586 shutdown_connections 2024-04-26T21:25:21.334 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:20.754+0000 7f5eaa828700 1 -- 172.21.15.46:0/4199348586 wait complete. 2024-04-26T21:25:21.334 INFO:teuthology.orchestra.run.smithi046.stdout:Enabling the dashboard module... 2024-04-26T21:25:21.952 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:21 smithi046 ceph-mon[29201]: from='mgr.14118 172.21.15.46:0/1110106616' entity='mgr.x' 2024-04-26T21:25:21.952 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:21 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/4199348586' entity='client.admin' 2024-04-26T21:25:22.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:22 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1189111039' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-26T21:25:22.804 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:22 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: ignoring --setuser ceph since I am not root 2024-04-26T21:25:22.804 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:22 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: ignoring --setgroup ceph since I am not root 2024-04-26T21:25:23.183 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:22 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:22.882+0000 7f94e8b88200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T21:25:23.183 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:23 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:23.075+0000 7f94e8b88200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T21:25:23.184 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:23 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:23.182+0000 7f94e8b88200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T21:25:23.544 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.764+0000 7fd107d83700 1 Processor -- start 2024-04-26T21:25:23.544 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.764+0000 7fd107d83700 1 -- start start 2024-04-26T21:25:23.544 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.764+0000 7fd107d83700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd100108d90 0x7fd1001091b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:23.544 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.764+0000 7fd107d83700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd100109780 con 0x7fd100108d90 2024-04-26T21:25:23.544 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.764+0000 7fd105b1f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd100108d90 0x7fd1001091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:23.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.764+0000 7fd105b1f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd100108d90 0x7fd1001091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37462/0 (socket says 172.21.15.46:37462) 2024-04-26T21:25:23.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.764+0000 7fd105b1f700 1 -- 172.21.15.46:0/2615870117 learned_addr learned my addr 172.21.15.46:0/2615870117 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:23.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.765+0000 7fd105b1f700 1 -- 172.21.15.46:0/2615870117 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd100109f90 con 0x7fd100108d90 2024-04-26T21:25:23.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.765+0000 7fd105b1f700 1 --2- 172.21.15.46:0/2615870117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd100108d90 0x7fd1001091b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc0098a0 tx=0x7fd0fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56345b56b5e14b2c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:23.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.765+0000 7fd104b1d700 1 -- 172.21.15.46:0/2615870117 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd0fc004160 con 0x7fd100108d90 2024-04-26T21:25:23.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.765+0000 7fd104b1d700 1 -- 172.21.15.46:0/2615870117 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0fc004300 con 0x7fd100108d90 2024-04-26T21:25:23.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.765+0000 7fd104b1d700 1 -- 172.21.15.46:0/2615870117 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd0fc011a20 con 0x7fd100108d90 2024-04-26T21:25:23.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.765+0000 7fd107d83700 1 -- 172.21.15.46:0/2615870117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd100108d90 msgr2=0x7fd1001091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:23.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.765+0000 7fd107d83700 1 --2- 172.21.15.46:0/2615870117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd100108d90 0x7fd1001091b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc0098a0 tx=0x7fd0fc009bb0 comp rx=0 tx=0).stop 2024-04-26T21:25:23.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.765+0000 7fd107d83700 1 -- 172.21.15.46:0/2615870117 shutdown_connections 2024-04-26T21:25:23.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.765+0000 7fd107d83700 1 --2- 172.21.15.46:0/2615870117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd100108d90 0x7fd1001091b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:23.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.765+0000 7fd107d83700 1 -- 172.21.15.46:0/2615870117 >> 172.21.15.46:0/2615870117 conn(0x7fd100104310 msgr2=0x7fd100106770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:23.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.766+0000 7fd107d83700 1 -- 172.21.15.46:0/2615870117 shutdown_connections 2024-04-26T21:25:23.545 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.766+0000 7fd107d83700 1 -- 172.21.15.46:0/2615870117 wait complete. 2024-04-26T21:25:23.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.766+0000 7fd107d83700 1 Processor -- start 2024-04-26T21:25:23.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.766+0000 7fd107d83700 1 -- start start 2024-04-26T21:25:23.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.766+0000 7fd107d83700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd100108d90 0x7fd10019cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:23.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.766+0000 7fd107d83700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd10010d340 con 0x7fd100108d90 2024-04-26T21:25:23.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.767+0000 7fd105b1f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd100108d90 0x7fd10019cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:23.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.767+0000 7fd105b1f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd100108d90 0x7fd10019cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37466/0 (socket says 172.21.15.46:37466) 2024-04-26T21:25:23.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.767+0000 7fd105b1f700 1 -- 172.21.15.46:0/1189111039 learned_addr learned my addr 172.21.15.46:0/1189111039 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:23.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.767+0000 7fd105b1f700 1 -- 172.21.15.46:0/1189111039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd10019d1c0 con 0x7fd100108d90 2024-04-26T21:25:23.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.767+0000 7fd105b1f700 1 --2- 172.21.15.46:0/1189111039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd100108d90 0x7fd10019cc80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc0119f0 tx=0x7fd0fc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:23.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.767+0000 7fd0f6ffd700 1 -- 172.21.15.46:0/1189111039 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd0fc0039e0 con 0x7fd100108d90 2024-04-26T21:25:23.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.767+0000 7fd0f6ffd700 1 -- 172.21.15.46:0/1189111039 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0fc01b8f0 con 0x7fd100108d90 2024-04-26T21:25:23.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.767+0000 7fd107d83700 1 -- 172.21.15.46:0/1189111039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd10019d450 con 0x7fd100108d90 2024-04-26T21:25:23.546 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.767+0000 7fd0f6ffd700 1 -- 172.21.15.46:0/1189111039 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd0fc0237a0 con 0x7fd100108d90 2024-04-26T21:25:23.547 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.767+0000 7fd107d83700 1 -- 172.21.15.46:0/1189111039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1001a0140 con 0x7fd100108d90 2024-04-26T21:25:23.547 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.768+0000 7fd0f6ffd700 1 -- 172.21.15.46:0/1189111039 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fd0fc01b440 con 0x7fd100108d90 2024-04-26T21:25:23.547 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.768+0000 7fd107d83700 1 -- 172.21.15.46:0/1189111039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd100064a80 con 0x7fd100108d90 2024-04-26T21:25:23.547 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.768+0000 7fd0f6ffd700 1 --2- 172.21.15.46:0/1189111039 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fd0ec039560 0x7fd0ec03ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:23.547 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.768+0000 7fd0f6ffd700 1 -- 172.21.15.46:0/1189111039 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd0fc04dec0 con 0x7fd100108d90 2024-04-26T21:25:23.547 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.768+0000 7fd10531e700 1 --2- 172.21.15.46:0/1189111039 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fd0ec039560 0x7fd0ec03ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:23.547 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.769+0000 7fd10531e700 1 --2- 172.21.15.46:0/1189111039 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fd0ec039560 0x7fd0ec03ba20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0006fd0 tx=0x7fd0f0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:23.547 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:21.774+0000 7fd0f6ffd700 1 -- 172.21.15.46:0/1189111039 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd0fc01f030 con 0x7fd100108d90 2024-04-26T21:25:23.547 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:22.058+0000 7fd107d83700 1 -- 172.21.15.46:0/1189111039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fd1001a06f0 con 0x7fd100108d90 2024-04-26T21:25:23.547 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:22.693+0000 7fd0f6ffd700 1 -- 172.21.15.46:0/1189111039 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fd0fc00be50 con 0x7fd100108d90 2024-04-26T21:25:23.547 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:22.695+0000 7fd0f6ffd700 1 -- 172.21.15.46:0/1189111039 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fd0fc02c910 con 0x7fd100108d90 2024-04-26T21:25:23.547 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:22.696+0000 7fd107d83700 1 -- 172.21.15.46:0/1189111039 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fd0ec039560 msgr2=0x7fd0ec03ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:23.548 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:22.696+0000 7fd107d83700 1 --2- 172.21.15.46:0/1189111039 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fd0ec039560 0x7fd0ec03ba20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0006fd0 tx=0x7fd0f0006e40 comp rx=0 tx=0).stop 2024-04-26T21:25:23.548 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:22.697+0000 7fd107d83700 1 -- 172.21.15.46:0/1189111039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd100108d90 msgr2=0x7fd10019cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:23.548 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:22.697+0000 7fd107d83700 1 --2- 172.21.15.46:0/1189111039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd100108d90 0x7fd10019cc80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc0119f0 tx=0x7fd0fc011ad0 comp rx=0 tx=0).stop 2024-04-26T21:25:23.548 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:22.697+0000 7fd107d83700 1 -- 172.21.15.46:0/1189111039 shutdown_connections 2024-04-26T21:25:23.548 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:22.697+0000 7fd107d83700 1 --2- 172.21.15.46:0/1189111039 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fd0ec039560 0x7fd0ec03ba20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:23.548 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:22.697+0000 7fd107d83700 1 --2- 172.21.15.46:0/1189111039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd100108d90 0x7fd10019cc80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:23.548 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:22.697+0000 7fd107d83700 1 -- 172.21.15.46:0/1189111039 >> 172.21.15.46:0/1189111039 conn(0x7fd100104310 msgr2=0x7fd100104dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:23.548 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:22.697+0000 7fd107d83700 1 -- 172.21.15.46:0/1189111039 shutdown_connections 2024-04-26T21:25:23.548 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:22.697+0000 7fd107d83700 1 -- 172.21.15.46:0/1189111039 wait complete. 2024-04-26T21:25:23.695 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:23 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:23.509+0000 7f94e8b88200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T21:25:23.695 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:23 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:23.601+0000 7f94e8b88200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T21:25:23.998 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:23 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1189111039' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-26T21:25:23.998 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:23 smithi046 ceph-mon[29201]: mgrmap e9: x(active, since 20s) 2024-04-26T21:25:24.909 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:24 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1165811637' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T21:25:24.909 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:24 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:24.872+0000 7f94e8b88200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T21:25:24.918 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-04-26T21:25:24.918 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-26T21:25:24.918 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T21:25:24.918 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-26T21:25:24.918 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T21:25:24.919 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-04-26T21:25:24.919 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.044+0000 7f2eb2023700 1 Processor -- start 2024-04-26T21:25:24.919 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.044+0000 7f2eb2023700 1 -- start start 2024-04-26T21:25:24.919 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.044+0000 7f2eb2023700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eac0751f0 0x7f2eac073650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:24.919 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.044+0000 7f2eb2023700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2eac073c20 con 0x7f2eac0751f0 2024-04-26T21:25:24.919 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.045+0000 7f2eb1021700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eac0751f0 0x7f2eac073650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:24.919 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.045+0000 7f2eb1021700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eac0751f0 0x7f2eac073650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37490/0 (socket says 172.21.15.46:37490) 2024-04-26T21:25:24.919 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.045+0000 7f2eb1021700 1 -- 172.21.15.46:0/2163524045 learned_addr learned my addr 172.21.15.46:0/2163524045 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:24.919 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.046+0000 7f2eb1021700 1 -- 172.21.15.46:0/2163524045 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eac073da0 con 0x7f2eac0751f0 2024-04-26T21:25:24.919 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.047+0000 7f2eb1021700 1 --2- 172.21.15.46:0/2163524045 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eac0751f0 0x7f2eac073650 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2ea40098a0 tx=0x7f2ea4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5f0f0d5a2e64a27 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:24.919 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.047+0000 7f2eabfff700 1 -- 172.21.15.46:0/2163524045 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ea4011bb0 con 0x7f2eac0751f0 2024-04-26T21:25:24.919 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.048+0000 7f2eabfff700 1 -- 172.21.15.46:0/2163524045 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ea40044e0 con 0x7f2eac0751f0 2024-04-26T21:25:24.919 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.049+0000 7f2eb2023700 1 -- 172.21.15.46:0/2163524045 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eac0751f0 msgr2=0x7f2eac073650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:24.919 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.049+0000 7f2eb2023700 1 --2- 172.21.15.46:0/2163524045 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eac0751f0 0x7f2eac073650 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2ea40098a0 tx=0x7f2ea4009bb0 comp rx=0 tx=0).stop 2024-04-26T21:25:24.920 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.049+0000 7f2eb2023700 1 -- 172.21.15.46:0/2163524045 shutdown_connections 2024-04-26T21:25:24.920 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.049+0000 7f2eb2023700 1 --2- 172.21.15.46:0/2163524045 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eac0751f0 0x7f2eac073650 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:24.920 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.049+0000 7f2eb2023700 1 -- 172.21.15.46:0/2163524045 >> 172.21.15.46:0/2163524045 conn(0x7f2eac06f010 msgr2=0x7f2eac071430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:24.920 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.050+0000 7f2eb2023700 1 -- 172.21.15.46:0/2163524045 shutdown_connections 2024-04-26T21:25:24.920 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.050+0000 7f2eb2023700 1 -- 172.21.15.46:0/2163524045 wait complete. 2024-04-26T21:25:24.920 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.051+0000 7f2eb2023700 1 Processor -- start 2024-04-26T21:25:24.920 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.051+0000 7f2eb2023700 1 -- start start 2024-04-26T21:25:24.920 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.051+0000 7f2eb2023700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eac083500 0x7f2eac083920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:24.920 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.051+0000 7f2eb2023700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2eac07e7b0 con 0x7f2eac083500 2024-04-26T21:25:24.920 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.052+0000 7f2eb1021700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eac083500 0x7f2eac083920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:24.920 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.052+0000 7f2eb1021700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eac083500 0x7f2eac083920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37494/0 (socket says 172.21.15.46:37494) 2024-04-26T21:25:24.920 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.052+0000 7f2eb1021700 1 -- 172.21.15.46:0/1165811637 learned_addr learned my addr 172.21.15.46:0/1165811637 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:24.920 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.053+0000 7f2eb1021700 1 -- 172.21.15.46:0/1165811637 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eac083e60 con 0x7f2eac083500 2024-04-26T21:25:24.921 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.054+0000 7f2eb1021700 1 --2- 172.21.15.46:0/1165811637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eac083500 0x7f2eac083920 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2ea400b130 tx=0x7f2ea40117e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:24.921 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.054+0000 7f2eaa7fc700 1 -- 172.21.15.46:0/1165811637 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ea4004000 con 0x7f2eac083500 2024-04-26T21:25:24.921 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.054+0000 7f2eb2023700 1 -- 172.21.15.46:0/1165811637 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eac0840f0 con 0x7f2eac083500 2024-04-26T21:25:24.921 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.055+0000 7f2eb2023700 1 -- 172.21.15.46:0/1165811637 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eac086c50 con 0x7f2eac083500 2024-04-26T21:25:24.921 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.055+0000 7f2eaa7fc700 1 -- 172.21.15.46:0/1165811637 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ea4004650 con 0x7f2eac083500 2024-04-26T21:25:24.921 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.055+0000 7f2eaa7fc700 1 -- 172.21.15.46:0/1165811637 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2ea401a7a0 con 0x7f2eac083500 2024-04-26T21:25:24.921 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.057+0000 7f2eaa7fc700 1 -- 172.21.15.46:0/1165811637 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f2ea40041a0 con 0x7f2eac083500 2024-04-26T21:25:24.921 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.057+0000 7f2eb2023700 1 -- 172.21.15.46:0/1165811637 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e98005360 con 0x7f2eac083500 2024-04-26T21:25:24.921 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.057+0000 7f2eaa7fc700 1 --2- 172.21.15.46:0/1165811637 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f2e94039610 0x7f2e9403bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:24.921 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.057+0000 7f2eaa7fc700 1 -- 172.21.15.46:0/1165811637 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2ea404d6c0 con 0x7f2eac083500 2024-04-26T21:25:24.921 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.057+0000 7f2eb0820700 1 -- 172.21.15.46:0/1165811637 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f2e94039610 msgr2=0x7f2e9403bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2053782984 2024-04-26T21:25:24.921 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.057+0000 7f2eb0820700 1 --2- 172.21.15.46:0/1165811637 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f2e94039610 0x7f2e9403bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T21:25:24.921 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.066+0000 7f2eaa7fc700 1 -- 172.21.15.46:0/1165811637 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2ea401e070 con 0x7f2eac083500 2024-04-26T21:25:24.921 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.258+0000 7f2eb0820700 1 -- 172.21.15.46:0/1165811637 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f2e94039610 msgr2=0x7f2e9403bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2053782984 2024-04-26T21:25:24.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.258+0000 7f2eb0820700 1 --2- 172.21.15.46:0/1165811637 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f2e94039610 0x7f2e9403bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T21:25:24.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.355+0000 7f2eb2023700 1 -- 172.21.15.46:0/1165811637 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f2e98005e60 con 0x7f2eac083500 2024-04-26T21:25:24.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.356+0000 7f2eaa7fc700 1 -- 172.21.15.46:0/1165811637 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f2ea4029340 con 0x7f2eac083500 2024-04-26T21:25:24.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.358+0000 7f2eb2023700 1 -- 172.21.15.46:0/1165811637 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f2e94039610 msgr2=0x7f2e9403bad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T21:25:24.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.358+0000 7f2eb2023700 1 --2- 172.21.15.46:0/1165811637 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f2e94039610 0x7f2e9403bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:24.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.358+0000 7f2eb2023700 1 -- 172.21.15.46:0/1165811637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eac083500 msgr2=0x7f2eac083920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:24.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.358+0000 7f2eb2023700 1 --2- 172.21.15.46:0/1165811637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eac083500 0x7f2eac083920 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2ea400b130 tx=0x7f2ea40117e0 comp rx=0 tx=0).stop 2024-04-26T21:25:24.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.358+0000 7f2eb2023700 1 -- 172.21.15.46:0/1165811637 shutdown_connections 2024-04-26T21:25:24.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.358+0000 7f2eb2023700 1 --2- 172.21.15.46:0/1165811637 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7f2e94039610 0x7f2e9403bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:24.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.358+0000 7f2eb2023700 1 --2- 172.21.15.46:0/1165811637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2eac083500 0x7f2eac083920 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:24.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.358+0000 7f2eb2023700 1 -- 172.21.15.46:0/1165811637 >> 172.21.15.46:0/1165811637 conn(0x7f2eac06f010 msgr2=0x7f2eac06f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:24.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.358+0000 7f2eb2023700 1 -- 172.21.15.46:0/1165811637 shutdown_connections 2024-04-26T21:25:24.922 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:24.358+0000 7f2eb2023700 1 -- 172.21.15.46:0/1165811637 wait complete. 2024-04-26T21:25:24.923 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for the mgr to restart... 2024-04-26T21:25:24.923 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mgr epoch 9... 2024-04-26T21:25:25.199 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:25 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:25.056+0000 7f94e8b88200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T21:25:25.543 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:25 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:25.237+0000 7f94e8b88200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T21:25:25.543 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:25 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:25.542+0000 7f94e8b88200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T21:25:25.803 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:25 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:25.640+0000 7f94e8b88200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T21:25:26.303 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:25 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:25.851+0000 7f94e8b88200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T21:25:26.303 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:26 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:26.051+0000 7f94e8b88200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T21:25:27.053 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:26 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:26.681+0000 7f94e8b88200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T21:25:27.432 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:27 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:27.119+0000 7f94e8b88200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T21:25:27.432 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:27 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:27.220+0000 7f94e8b88200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T21:25:27.432 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:27 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:27.430+0000 7f94e8b88200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T21:25:28.303 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:27 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:27.975+0000 7f94e8b88200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T21:25:28.303 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:28 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:28.071+0000 7f94e8b88200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T21:25:28.681 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:28 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:28.323+0000 7f94e8b88200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T21:25:28.681 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:28 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:28.461+0000 7f94e8b88200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T21:25:28.681 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:28 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:28.560+0000 7f94e8b88200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T21:25:28.681 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:28 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:28.679+0000 7f94e8b88200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T21:25:29.303 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:28 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:28.992+0000 7f94e8b88200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T21:25:29.900 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:29 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:29.621+0000 7f94e8b88200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T21:25:29.900 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:25:29 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:25:29.845+0000 7f94e8b88200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T21:25:30.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:29 smithi046 ceph-mon[29201]: Active manager daemon x restarted 2024-04-26T21:25:30.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:29 smithi046 ceph-mon[29201]: Activating manager daemon x 2024-04-26T21:25:30.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:29 smithi046 ceph-mon[29201]: osdmap e3: 0 total, 0 up, 0 in 2024-04-26T21:25:30.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:29 smithi046 ceph-mon[29201]: mgrmap e10: x(active, starting, since 0.00323664s) 2024-04-26T21:25:30.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:29 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:25:30.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:29 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T21:25:30.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:29 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T21:25:30.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:29 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T21:25:30.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:29 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T21:25:30.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:29 smithi046 ceph-mon[29201]: Manager daemon x is now available 2024-04-26T21:25:30.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:29 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:30.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:29 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T21:25:31.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:30 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T21:25:31.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:30 smithi046 ceph-mon[29201]: [26/Apr/2024:21:25:30] ENGINE Bus STARTING 2024-04-26T21:25:31.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:30 smithi046 ceph-mon[29201]: [26/Apr/2024:21:25:30] ENGINE Serving on https://172.21.15.46:7150 2024-04-26T21:25:31.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:30 smithi046 ceph-mon[29201]: mgrmap e11: x(active, since 1.00615s) 2024-04-26T21:25:31.695 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-04-26T21:25:31.696 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-26T21:25:31.696 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T21:25:31.696 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-04-26T21:25:31.696 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.414+0000 7fc04696a700 1 Processor -- start 2024-04-26T21:25:31.696 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.414+0000 7fc04696a700 1 -- start start 2024-04-26T21:25:31.696 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.414+0000 7fc04696a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc040109860 0x7fc04010bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:31.696 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.414+0000 7fc04696a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc04007d680 con 0x7fc040109860 2024-04-26T21:25:31.696 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.414+0000 7fc03ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc040109860 0x7fc04010bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:31.696 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.414+0000 7fc03ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc040109860 0x7fc04010bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37506/0 (socket says 172.21.15.46:37506) 2024-04-26T21:25:31.696 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.414+0000 7fc03ffff700 1 -- 172.21.15.46:0/2715921473 learned_addr learned my addr 172.21.15.46:0/2715921473 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:31.696 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.415+0000 7fc03ffff700 1 -- 172.21.15.46:0/2715921473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc04010c190 con 0x7fc040109860 2024-04-26T21:25:31.697 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.415+0000 7fc03ffff700 1 --2- 172.21.15.46:0/2715921473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc040109860 0x7fc04010bc50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc0300098a0 tx=0x7fc030009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aade3d329d91448e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:31.697 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.415+0000 7fc03effd700 1 -- 172.21.15.46:0/2715921473 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc030004160 con 0x7fc040109860 2024-04-26T21:25:31.697 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.415+0000 7fc03effd700 1 -- 172.21.15.46:0/2715921473 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc030004300 con 0x7fc040109860 2024-04-26T21:25:31.697 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.415+0000 7fc03effd700 1 -- 172.21.15.46:0/2715921473 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc030011a20 con 0x7fc040109860 2024-04-26T21:25:31.697 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.416+0000 7fc04696a700 1 -- 172.21.15.46:0/2715921473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc040109860 msgr2=0x7fc04010bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:31.697 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.416+0000 7fc04696a700 1 --2- 172.21.15.46:0/2715921473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc040109860 0x7fc04010bc50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc0300098a0 tx=0x7fc030009bb0 comp rx=0 tx=0).stop 2024-04-26T21:25:31.697 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.416+0000 7fc04696a700 1 -- 172.21.15.46:0/2715921473 shutdown_connections 2024-04-26T21:25:31.697 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.416+0000 7fc04696a700 1 --2- 172.21.15.46:0/2715921473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc040109860 0x7fc04010bc50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:31.697 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.416+0000 7fc04696a700 1 -- 172.21.15.46:0/2715921473 >> 172.21.15.46:0/2715921473 conn(0x7fc040103230 msgr2=0x7fc040105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:31.697 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.416+0000 7fc04696a700 1 -- 172.21.15.46:0/2715921473 shutdown_connections 2024-04-26T21:25:31.697 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.416+0000 7fc04696a700 1 -- 172.21.15.46:0/2715921473 wait complete. 2024-04-26T21:25:31.697 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.417+0000 7fc04696a700 1 Processor -- start 2024-04-26T21:25:31.698 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.417+0000 7fc04696a700 1 -- start start 2024-04-26T21:25:31.698 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.417+0000 7fc04696a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc040109860 0x7fc04019cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:31.698 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.417+0000 7fc04696a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc04010d130 con 0x7fc040109860 2024-04-26T21:25:31.698 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.417+0000 7fc03ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc040109860 0x7fc04019cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:31.698 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.417+0000 7fc03ffff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc040109860 0x7fc04019cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37514/0 (socket says 172.21.15.46:37514) 2024-04-26T21:25:31.698 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.417+0000 7fc03ffff700 1 -- 172.21.15.46:0/2852920015 learned_addr learned my addr 172.21.15.46:0/2852920015 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:31.698 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.417+0000 7fc03ffff700 1 -- 172.21.15.46:0/2852920015 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc04019d190 con 0x7fc040109860 2024-04-26T21:25:31.698 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.418+0000 7fc03ffff700 1 --2- 172.21.15.46:0/2852920015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc040109860 0x7fc04019cc50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc030004660 tx=0x7fc030011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:31.698 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.418+0000 7fc03d7fa700 1 -- 172.21.15.46:0/2852920015 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc030003aa0 con 0x7fc040109860 2024-04-26T21:25:31.698 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.418+0000 7fc03d7fa700 1 -- 172.21.15.46:0/2852920015 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc03001b8f0 con 0x7fc040109860 2024-04-26T21:25:31.698 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.418+0000 7fc04696a700 1 -- 172.21.15.46:0/2852920015 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc04019d420 con 0x7fc040109860 2024-04-26T21:25:31.699 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.418+0000 7fc03d7fa700 1 -- 172.21.15.46:0/2852920015 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc030023900 con 0x7fc040109860 2024-04-26T21:25:31.699 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.418+0000 7fc04696a700 1 -- 172.21.15.46:0/2852920015 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0401a0110 con 0x7fc040109860 2024-04-26T21:25:31.699 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.419+0000 7fc03d7fa700 1 -- 172.21.15.46:0/2852920015 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fc030023b70 con 0x7fc040109860 2024-04-26T21:25:31.699 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.419+0000 7fc03d7fa700 1 --2- 172.21.15.46:0/2852920015 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fc0280396b0 0x7fc02803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:31.699 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.419+0000 7fc04696a700 1 -- 172.21.15.46:0/2852920015 --> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc04006a540 con 0x7fc0280396b0 2024-04-26T21:25:31.699 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.419+0000 7fc03d7fa700 1 -- 172.21.15.46:0/2852920015 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc03004eee0 con 0x7fc040109860 2024-04-26T21:25:31.699 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.419+0000 7fc03f7fe700 1 -- 172.21.15.46:0/2852920015 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fc0280396b0 msgr2=0x7fc02803bb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2053782984 2024-04-26T21:25:31.699 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.419+0000 7fc03f7fe700 1 --2- 172.21.15.46:0/2852920015 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fc0280396b0 0x7fc02803bb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T21:25:31.699 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.620+0000 7fc03f7fe700 1 -- 172.21.15.46:0/2852920015 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fc0280396b0 msgr2=0x7fc02803bb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2053782984 2024-04-26T21:25:31.699 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:25.620+0000 7fc03f7fe700 1 --2- 172.21.15.46:0/2852920015 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fc0280396b0 0x7fc02803bb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T21:25:31.699 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:26.021+0000 7fc03f7fe700 1 -- 172.21.15.46:0/2852920015 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fc0280396b0 msgr2=0x7fc02803bb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2053782984 2024-04-26T21:25:31.700 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:26.021+0000 7fc03f7fe700 1 --2- 172.21.15.46:0/2852920015 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fc0280396b0 0x7fc02803bb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T21:25:31.700 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:26.822+0000 7fc03f7fe700 1 -- 172.21.15.46:0/2852920015 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fc0280396b0 msgr2=0x7fc02803bb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2053782984 2024-04-26T21:25:31.700 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:26.822+0000 7fc03f7fe700 1 --2- 172.21.15.46:0/2852920015 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fc0280396b0 0x7fc02803bb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T21:25:31.700 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:28.423+0000 7fc03f7fe700 1 -- 172.21.15.46:0/2852920015 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fc0280396b0 msgr2=0x7fc02803bb70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2053782984 2024-04-26T21:25:31.700 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:28.423+0000 7fc03f7fe700 1 --2- 172.21.15.46:0/2852920015 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fc0280396b0 0x7fc02803bb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T21:25:31.700 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:29.849+0000 7fc03d7fa700 1 -- 172.21.15.46:0/2852920015 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7fc03001baa0 con 0x7fc040109860 2024-04-26T21:25:31.700 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:29.849+0000 7fc03d7fa700 1 -- 172.21.15.46:0/2852920015 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fc0280396b0 msgr2=0x7fc02803bb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T21:25:31.700 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:29.849+0000 7fc03d7fa700 1 --2- 172.21.15.46:0/2852920015 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fc0280396b0 0x7fc02803bb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:31.700 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:30.851+0000 7fc03d7fa700 1 -- 172.21.15.46:0/2852920015 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7fc03002cc90 con 0x7fc040109860 2024-04-26T21:25:31.700 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:30.851+0000 7fc03d7fa700 1 --2- 172.21.15.46:0/2852920015 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fc02803cf40 0x7fc02803f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:31.700 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:30.851+0000 7fc03d7fa700 1 -- 172.21.15.46:0/2852920015 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc030047f40 con 0x7fc02803cf40 2024-04-26T21:25:31.700 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:30.852+0000 7fc03f7fe700 1 --2- 172.21.15.46:0/2852920015 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fc02803cf40 0x7fc02803f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:31.700 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:30.852+0000 7fc03f7fe700 1 --2- 172.21.15.46:0/2852920015 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fc02803cf40 0x7fc02803f330 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc034003a10 tx=0x7fc0340092b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:31.700 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:30.853+0000 7fc03d7fa700 1 -- 172.21.15.46:0/2852920015 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7fc030047f40 con 0x7fc02803cf40 2024-04-26T21:25:31.700 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:30.861+0000 7fc04696a700 1 -- 172.21.15.46:0/2852920015 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fc0401043e0 con 0x7fc02803cf40 2024-04-26T21:25:31.700 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:30.861+0000 7fc03d7fa700 1 -- 172.21.15.46:0/2852920015 <== mgr.14150 v2:172.21.15.46:6800/4004175376 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fc0401043e0 con 0x7fc02803cf40 2024-04-26T21:25:31.701 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:30.862+0000 7fc04696a700 1 -- 172.21.15.46:0/2852920015 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fc02803cf40 msgr2=0x7fc02803f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:31.701 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:30.862+0000 7fc04696a700 1 --2- 172.21.15.46:0/2852920015 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fc02803cf40 0x7fc02803f330 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc034003a10 tx=0x7fc0340092b0 comp rx=0 tx=0).stop 2024-04-26T21:25:31.701 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:30.862+0000 7fc04696a700 1 -- 172.21.15.46:0/2852920015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc040109860 msgr2=0x7fc04019cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:31.701 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:30.862+0000 7fc04696a700 1 --2- 172.21.15.46:0/2852920015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc040109860 0x7fc04019cc50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc030004660 tx=0x7fc030011aa0 comp rx=0 tx=0).stop 2024-04-26T21:25:31.701 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:30.862+0000 7fc04696a700 1 -- 172.21.15.46:0/2852920015 shutdown_connections 2024-04-26T21:25:31.701 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:30.862+0000 7fc04696a700 1 --2- 172.21.15.46:0/2852920015 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fc02803cf40 0x7fc02803f330 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:31.701 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:30.862+0000 7fc04696a700 1 --2- 172.21.15.46:0/2852920015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc040109860 0x7fc04019cc50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:31.701 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:30.862+0000 7fc04696a700 1 --2- 172.21.15.46:0/2852920015 >> [v2:172.21.15.46:6800/2053782984,v1:172.21.15.46:6801/2053782984] conn(0x7fc0280396b0 0x7fc02803bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:31.701 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:30.862+0000 7fc04696a700 1 -- 172.21.15.46:0/2852920015 >> 172.21.15.46:0/2852920015 conn(0x7fc040103230 msgr2=0x7fc040103cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:31.701 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:30.862+0000 7fc04696a700 1 -- 172.21.15.46:0/2852920015 shutdown_connections 2024-04-26T21:25:31.701 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:30.862+0000 7fc04696a700 1 -- 172.21.15.46:0/2852920015 wait complete. 2024-04-26T21:25:31.701 INFO:teuthology.orchestra.run.smithi046.stdout:mgr epoch 9 is available 2024-04-26T21:25:31.701 INFO:teuthology.orchestra.run.smithi046.stdout:Generating a dashboard self-signed certificate... 2024-04-26T21:25:31.951 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:31 smithi046 ceph-mon[29201]: [26/Apr/2024:21:25:30] ENGINE Serving on http://172.21.15.46:8765 2024-04-26T21:25:31.951 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:31 smithi046 ceph-mon[29201]: [26/Apr/2024:21:25:30] ENGINE Bus STARTED 2024-04-26T21:25:31.951 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:31 smithi046 ceph-mon[29201]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T21:25:31.951 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:31 smithi046 ceph-mon[29201]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T21:25:33.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:32 smithi046 ceph-mon[29201]: mgrmap e12: x(active, since 2s) 2024-04-26T21:25:33.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:32 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:25:33.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:32 smithi046 ceph-mon[29201]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:25:33.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:32 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:25:33.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:32 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:25:33.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-26T21:25:33.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.180+0000 7ff84cf74700 1 Processor -- start 2024-04-26T21:25:33.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.180+0000 7ff84cf74700 1 -- start start 2024-04-26T21:25:33.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.180+0000 7ff84cf74700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff848108d70 0x7ff848109190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:33.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.180+0000 7ff84cf74700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff848109760 con 0x7ff848108d70 2024-04-26T21:25:33.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.181+0000 7ff84659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff848108d70 0x7ff848109190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:33.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.181+0000 7ff84659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff848108d70 0x7ff848109190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57732/0 (socket says 172.21.15.46:57732) 2024-04-26T21:25:33.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.181+0000 7ff84659c700 1 -- 172.21.15.46:0/1927186037 learned_addr learned my addr 172.21.15.46:0/1927186037 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:33.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.181+0000 7ff84659c700 1 -- 172.21.15.46:0/1927186037 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff848109f70 con 0x7ff848108d70 2024-04-26T21:25:33.233 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.181+0000 7ff84659c700 1 --2- 172.21.15.46:0/1927186037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff848108d70 0x7ff848109190 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff8380098a0 tx=0x7ff838009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba328c7d3083a0f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:33.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.181+0000 7ff84559a700 1 -- 172.21.15.46:0/1927186037 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff838004160 con 0x7ff848108d70 2024-04-26T21:25:33.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.181+0000 7ff84559a700 1 -- 172.21.15.46:0/1927186037 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff838004300 con 0x7ff848108d70 2024-04-26T21:25:33.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.182+0000 7ff84559a700 1 -- 172.21.15.46:0/1927186037 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff838011a20 con 0x7ff848108d70 2024-04-26T21:25:33.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.182+0000 7ff84cf74700 1 -- 172.21.15.46:0/1927186037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff848108d70 msgr2=0x7ff848109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:33.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.182+0000 7ff84cf74700 1 --2- 172.21.15.46:0/1927186037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff848108d70 0x7ff848109190 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff8380098a0 tx=0x7ff838009bb0 comp rx=0 tx=0).stop 2024-04-26T21:25:33.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.183+0000 7ff84cf74700 1 -- 172.21.15.46:0/1927186037 shutdown_connections 2024-04-26T21:25:33.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.183+0000 7ff84cf74700 1 --2- 172.21.15.46:0/1927186037 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff848108d70 0x7ff848109190 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:33.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.183+0000 7ff84cf74700 1 -- 172.21.15.46:0/1927186037 >> 172.21.15.46:0/1927186037 conn(0x7ff8481042d0 msgr2=0x7ff848106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:33.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.183+0000 7ff84cf74700 1 -- 172.21.15.46:0/1927186037 shutdown_connections 2024-04-26T21:25:33.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.183+0000 7ff84cf74700 1 -- 172.21.15.46:0/1927186037 wait complete. 2024-04-26T21:25:33.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.183+0000 7ff84cf74700 1 Processor -- start 2024-04-26T21:25:33.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.184+0000 7ff84cf74700 1 -- start start 2024-04-26T21:25:33.234 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.184+0000 7ff84cf74700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff848108d70 0x7ff84819ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:33.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.184+0000 7ff84cf74700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff84810d320 con 0x7ff848108d70 2024-04-26T21:25:33.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.184+0000 7ff84659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff848108d70 0x7ff84819ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:33.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.184+0000 7ff84659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff848108d70 0x7ff84819ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57742/0 (socket says 172.21.15.46:57742) 2024-04-26T21:25:33.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.184+0000 7ff84659c700 1 -- 172.21.15.46:0/2422183253 learned_addr learned my addr 172.21.15.46:0/2422183253 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:33.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.184+0000 7ff84659c700 1 -- 172.21.15.46:0/2422183253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff84819d200 con 0x7ff848108d70 2024-04-26T21:25:33.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.184+0000 7ff84659c700 1 --2- 172.21.15.46:0/2422183253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff848108d70 0x7ff84819ccc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff838000c00 tx=0x7ff838011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:33.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.185+0000 7ff8377fe700 1 -- 172.21.15.46:0/2422183253 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff838003aa0 con 0x7ff848108d70 2024-04-26T21:25:33.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.185+0000 7ff84cf74700 1 -- 172.21.15.46:0/2422183253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff84819d490 con 0x7ff848108d70 2024-04-26T21:25:33.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.185+0000 7ff84cf74700 1 -- 172.21.15.46:0/2422183253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8481a0180 con 0x7ff848108d70 2024-04-26T21:25:33.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.185+0000 7ff8377fe700 1 -- 172.21.15.46:0/2422183253 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff83801b8f0 con 0x7ff848108d70 2024-04-26T21:25:33.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.185+0000 7ff8377fe700 1 -- 172.21.15.46:0/2422183253 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff838023890 con 0x7ff848108d70 2024-04-26T21:25:33.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.186+0000 7ff8377fe700 1 -- 172.21.15.46:0/2422183253 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7ff83802c3f0 con 0x7ff848108d70 2024-04-26T21:25:33.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.186+0000 7ff8377fe700 1 --2- 172.21.15.46:0/2422183253 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ff830039660 0x7ff83003bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:33.235 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.186+0000 7ff84cf74700 1 -- 172.21.15.46:0/2422183253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff828005360 con 0x7ff848108d70 2024-04-26T21:25:33.236 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.186+0000 7ff8377fe700 1 -- 172.21.15.46:0/2422183253 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff83804e0e0 con 0x7ff848108d70 2024-04-26T21:25:33.236 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.186+0000 7ff845d9b700 1 --2- 172.21.15.46:0/2422183253 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ff830039660 0x7ff83003bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:33.236 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.187+0000 7ff845d9b700 1 --2- 172.21.15.46:0/2422183253 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ff830039660 0x7ff83003bb20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff83c006fd0 tx=0x7ff83c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:33.236 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.191+0000 7ff8377fe700 1 -- 172.21.15.46:0/2422183253 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff83801f030 con 0x7ff848108d70 2024-04-26T21:25:33.236 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.424+0000 7ff84cf74700 1 -- 172.21.15.46:0/2422183253 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7ff828000bf0 con 0x7ff830039660 2024-04-26T21:25:33.236 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.526+0000 7ff8377fe700 1 -- 172.21.15.46:0/2422183253 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7ff828000bf0 con 0x7ff830039660 2024-04-26T21:25:33.236 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.528+0000 7ff84cf74700 1 -- 172.21.15.46:0/2422183253 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ff830039660 msgr2=0x7ff83003bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:33.236 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.528+0000 7ff84cf74700 1 --2- 172.21.15.46:0/2422183253 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ff830039660 0x7ff83003bb20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff83c006fd0 tx=0x7ff83c006e40 comp rx=0 tx=0).stop 2024-04-26T21:25:33.236 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.528+0000 7ff84cf74700 1 -- 172.21.15.46:0/2422183253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff848108d70 msgr2=0x7ff84819ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:33.236 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.528+0000 7ff84cf74700 1 --2- 172.21.15.46:0/2422183253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff848108d70 0x7ff84819ccc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff838000c00 tx=0x7ff838011aa0 comp rx=0 tx=0).stop 2024-04-26T21:25:33.236 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.528+0000 7ff84cf74700 1 -- 172.21.15.46:0/2422183253 shutdown_connections 2024-04-26T21:25:33.236 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.528+0000 7ff84cf74700 1 --2- 172.21.15.46:0/2422183253 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ff830039660 0x7ff83003bb20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:33.236 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.528+0000 7ff84cf74700 1 --2- 172.21.15.46:0/2422183253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff848108d70 0x7ff84819ccc0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:33.236 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.528+0000 7ff84cf74700 1 -- 172.21.15.46:0/2422183253 >> 172.21.15.46:0/2422183253 conn(0x7ff8481042d0 msgr2=0x7ff848104e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:33.237 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.529+0000 7ff84cf74700 1 -- 172.21.15.46:0/2422183253 shutdown_connections 2024-04-26T21:25:33.237 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:32.529+0000 7ff84cf74700 1 -- 172.21.15.46:0/2422183253 wait complete. 2024-04-26T21:25:33.237 INFO:teuthology.orchestra.run.smithi046.stdout:Creating initial admin user... 2024-04-26T21:25:34.791 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$IuAXkMkfleyw0foR9msF5.846Lv.BZeR2ZKCw9DSg8GiM3wGDPrzO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714166734, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-26T21:25:34.791 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.702+0000 7ff5e135f700 1 Processor -- start 2024-04-26T21:25:34.791 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.702+0000 7ff5e135f700 1 -- start start 2024-04-26T21:25:34.791 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.702+0000 7ff5e135f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc1098a0 0x7ff5dc10bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:34.791 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.702+0000 7ff5e135f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff5dc07d650 con 0x7ff5dc1098a0 2024-04-26T21:25:34.791 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.702+0000 7ff5daffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc1098a0 0x7ff5dc10bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:34.791 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.702+0000 7ff5daffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc1098a0 0x7ff5dc10bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57744/0 (socket says 172.21.15.46:57744) 2024-04-26T21:25:34.791 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.702+0000 7ff5daffd700 1 -- 172.21.15.46:0/1859458803 learned_addr learned my addr 172.21.15.46:0/1859458803 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:34.791 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.702+0000 7ff5daffd700 1 -- 172.21.15.46:0/1859458803 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5dc10c1d0 con 0x7ff5dc1098a0 2024-04-26T21:25:34.791 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.703+0000 7ff5daffd700 1 --2- 172.21.15.46:0/1859458803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc1098a0 0x7ff5dc10bc90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc0098a0 tx=0x7ff5cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb0967ac80c89e1e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:34.791 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.703+0000 7ff5d9ffb700 1 -- 172.21.15.46:0/1859458803 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5cc004160 con 0x7ff5dc1098a0 2024-04-26T21:25:34.792 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.703+0000 7ff5d9ffb700 1 -- 172.21.15.46:0/1859458803 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5cc004300 con 0x7ff5dc1098a0 2024-04-26T21:25:34.792 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.703+0000 7ff5d9ffb700 1 -- 172.21.15.46:0/1859458803 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5cc011a20 con 0x7ff5dc1098a0 2024-04-26T21:25:34.792 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.703+0000 7ff5e135f700 1 -- 172.21.15.46:0/1859458803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc1098a0 msgr2=0x7ff5dc10bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:34.792 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.703+0000 7ff5e135f700 1 --2- 172.21.15.46:0/1859458803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc1098a0 0x7ff5dc10bc90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc0098a0 tx=0x7ff5cc009bb0 comp rx=0 tx=0).stop 2024-04-26T21:25:34.792 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.703+0000 7ff5e135f700 1 -- 172.21.15.46:0/1859458803 shutdown_connections 2024-04-26T21:25:34.792 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.703+0000 7ff5e135f700 1 --2- 172.21.15.46:0/1859458803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc1098a0 0x7ff5dc10bc90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:34.792 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.703+0000 7ff5e135f700 1 -- 172.21.15.46:0/1859458803 >> 172.21.15.46:0/1859458803 conn(0x7ff5dc1031f0 msgr2=0x7ff5dc105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:34.792 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.704+0000 7ff5e135f700 1 -- 172.21.15.46:0/1859458803 shutdown_connections 2024-04-26T21:25:34.792 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.704+0000 7ff5e135f700 1 -- 172.21.15.46:0/1859458803 wait complete. 2024-04-26T21:25:34.792 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.704+0000 7ff5e135f700 1 Processor -- start 2024-04-26T21:25:34.792 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.705+0000 7ff5e135f700 1 -- start start 2024-04-26T21:25:34.792 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.705+0000 7ff5e135f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc1098a0 0x7ff5dc19cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:34.792 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.705+0000 7ff5e135f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff5dc10d110 con 0x7ff5dc1098a0 2024-04-26T21:25:34.793 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.705+0000 7ff5daffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc1098a0 0x7ff5dc19cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:34.793 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.705+0000 7ff5daffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc1098a0 0x7ff5dc19cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57758/0 (socket says 172.21.15.46:57758) 2024-04-26T21:25:34.793 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.705+0000 7ff5daffd700 1 -- 172.21.15.46:0/3922761860 learned_addr learned my addr 172.21.15.46:0/3922761860 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:34.793 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.705+0000 7ff5daffd700 1 -- 172.21.15.46:0/3922761860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5dc19d1b0 con 0x7ff5dc1098a0 2024-04-26T21:25:34.793 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.706+0000 7ff5daffd700 1 --2- 172.21.15.46:0/3922761860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc1098a0 0x7ff5dc19cc70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc004660 tx=0x7ff5cc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:34.793 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.706+0000 7ff5c3fff700 1 -- 172.21.15.46:0/3922761860 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5cc003aa0 con 0x7ff5dc1098a0 2024-04-26T21:25:34.793 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.706+0000 7ff5c3fff700 1 -- 172.21.15.46:0/3922761860 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5cc01b8f0 con 0x7ff5dc1098a0 2024-04-26T21:25:34.793 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.706+0000 7ff5e135f700 1 -- 172.21.15.46:0/3922761860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5dc19d440 con 0x7ff5dc1098a0 2024-04-26T21:25:34.793 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.706+0000 7ff5c3fff700 1 -- 172.21.15.46:0/3922761860 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5cc023900 con 0x7ff5dc1098a0 2024-04-26T21:25:34.793 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.706+0000 7ff5e135f700 1 -- 172.21.15.46:0/3922761860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5dc1a0130 con 0x7ff5dc1098a0 2024-04-26T21:25:34.793 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.707+0000 7ff5e135f700 1 -- 172.21.15.46:0/3922761860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5dc064a80 con 0x7ff5dc1098a0 2024-04-26T21:25:34.793 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.707+0000 7ff5c3fff700 1 -- 172.21.15.46:0/3922761860 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7ff5cc023b70 con 0x7ff5dc1098a0 2024-04-26T21:25:34.793 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.707+0000 7ff5c3fff700 1 --2- 172.21.15.46:0/3922761860 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ff5c4039660 0x7ff5c403bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:34.793 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.707+0000 7ff5c3fff700 1 -- 172.21.15.46:0/3922761860 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff5cc04e080 con 0x7ff5dc1098a0 2024-04-26T21:25:34.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.707+0000 7ff5da7fc700 1 --2- 172.21.15.46:0/3922761860 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ff5c4039660 0x7ff5c403bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:34.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.708+0000 7ff5da7fc700 1 --2- 172.21.15.46:0/3922761860 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ff5c4039660 0x7ff5c403bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0006fd0 tx=0x7ff5d0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:34.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.712+0000 7ff5c3fff700 1 -- 172.21.15.46:0/3922761860 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff5cc01f030 con 0x7ff5dc1098a0 2024-04-26T21:25:34.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:33.939+0000 7ff5e135f700 1 -- 172.21.15.46:0/3922761860 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7ff5dc1a03c0 con 0x7ff5c4039660 2024-04-26T21:25:34.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:34.186+0000 7ff5c3fff700 1 -- 172.21.15.46:0/3922761860 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7ff5dc1a03c0 con 0x7ff5c4039660 2024-04-26T21:25:34.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:34.189+0000 7ff5e135f700 1 -- 172.21.15.46:0/3922761860 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ff5c4039660 msgr2=0x7ff5c403bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:34.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:34.189+0000 7ff5e135f700 1 --2- 172.21.15.46:0/3922761860 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ff5c4039660 0x7ff5c403bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0006fd0 tx=0x7ff5d0006e40 comp rx=0 tx=0).stop 2024-04-26T21:25:34.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:34.189+0000 7ff5e135f700 1 -- 172.21.15.46:0/3922761860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc1098a0 msgr2=0x7ff5dc19cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:34.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:34.189+0000 7ff5e135f700 1 --2- 172.21.15.46:0/3922761860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc1098a0 0x7ff5dc19cc70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc004660 tx=0x7ff5cc011aa0 comp rx=0 tx=0).stop 2024-04-26T21:25:34.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:34.189+0000 7ff5e135f700 1 -- 172.21.15.46:0/3922761860 shutdown_connections 2024-04-26T21:25:34.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:34.189+0000 7ff5e135f700 1 --2- 172.21.15.46:0/3922761860 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ff5c4039660 0x7ff5c403bb20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:34.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:34.189+0000 7ff5e135f700 1 --2- 172.21.15.46:0/3922761860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5dc1098a0 0x7ff5dc19cc70 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:34.794 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:34.189+0000 7ff5e135f700 1 -- 172.21.15.46:0/3922761860 >> 172.21.15.46:0/3922761860 conn(0x7ff5dc1031f0 msgr2=0x7ff5dc103cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:34.795 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:34.189+0000 7ff5e135f700 1 -- 172.21.15.46:0/3922761860 shutdown_connections 2024-04-26T21:25:34.795 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:34.189+0000 7ff5e135f700 1 -- 172.21.15.46:0/3922761860 wait complete. 2024-04-26T21:25:34.795 INFO:teuthology.orchestra.run.smithi046.stdout:Fetching dashboard port number... 2024-04-26T21:25:34.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:34 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:25:34.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:34 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:25:35.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:35 smithi046 ceph-mon[29201]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:25:35.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:35 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1927040270' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-26T21:25:36.191 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 8443 2024-04-26T21:25:36.192 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.238+0000 7fe7f5f08700 1 Processor -- start 2024-04-26T21:25:36.192 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.238+0000 7fe7f5f08700 1 -- start start 2024-04-26T21:25:36.192 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.238+0000 7fe7f5f08700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f0107c50 0x7fe7f0108070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:36.192 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.238+0000 7fe7f5f08700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f0108640 con 0x7fe7f0107c50 2024-04-26T21:25:36.192 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.238+0000 7fe7ef7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f0107c50 0x7fe7f0108070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:36.192 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.239+0000 7fe7ef7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f0107c50 0x7fe7f0108070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57764/0 (socket says 172.21.15.46:57764) 2024-04-26T21:25:36.192 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.239+0000 7fe7ef7fe700 1 -- 172.21.15.46:0/2659394805 learned_addr learned my addr 172.21.15.46:0/2659394805 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:36.192 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.239+0000 7fe7ef7fe700 1 -- 172.21.15.46:0/2659394805 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7f0108e50 con 0x7fe7f0107c50 2024-04-26T21:25:36.192 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.239+0000 7fe7ef7fe700 1 --2- 172.21.15.46:0/2659394805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f0107c50 0x7fe7f0108070 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe7e00098a0 tx=0x7fe7e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=385511b250a6322e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:36.192 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.239+0000 7fe7ee7fc700 1 -- 172.21.15.46:0/2659394805 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7e0004160 con 0x7fe7f0107c50 2024-04-26T21:25:36.192 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.239+0000 7fe7ee7fc700 1 -- 172.21.15.46:0/2659394805 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7e0004300 con 0x7fe7f0107c50 2024-04-26T21:25:36.192 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.239+0000 7fe7ee7fc700 1 -- 172.21.15.46:0/2659394805 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7e0011a20 con 0x7fe7f0107c50 2024-04-26T21:25:36.192 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.240+0000 7fe7f5f08700 1 -- 172.21.15.46:0/2659394805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f0107c50 msgr2=0x7fe7f0108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:36.193 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.240+0000 7fe7f5f08700 1 --2- 172.21.15.46:0/2659394805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f0107c50 0x7fe7f0108070 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe7e00098a0 tx=0x7fe7e0009bb0 comp rx=0 tx=0).stop 2024-04-26T21:25:36.193 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.240+0000 7fe7f5f08700 1 -- 172.21.15.46:0/2659394805 shutdown_connections 2024-04-26T21:25:36.193 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.240+0000 7fe7f5f08700 1 --2- 172.21.15.46:0/2659394805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f0107c50 0x7fe7f0108070 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:36.193 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.240+0000 7fe7f5f08700 1 -- 172.21.15.46:0/2659394805 >> 172.21.15.46:0/2659394805 conn(0x7fe7f01031f0 msgr2=0x7fe7f0105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:36.193 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.240+0000 7fe7f5f08700 1 -- 172.21.15.46:0/2659394805 shutdown_connections 2024-04-26T21:25:36.193 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.240+0000 7fe7f5f08700 1 -- 172.21.15.46:0/2659394805 wait complete. 2024-04-26T21:25:36.193 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.240+0000 7fe7f5f08700 1 Processor -- start 2024-04-26T21:25:36.193 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.241+0000 7fe7f5f08700 1 -- start start 2024-04-26T21:25:36.193 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.241+0000 7fe7f5f08700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f0107c50 0x7fe7f019cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:36.193 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.241+0000 7fe7f5f08700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe7f010c200 con 0x7fe7f0107c50 2024-04-26T21:25:36.193 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.241+0000 7fe7ef7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f0107c50 0x7fe7f019cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:36.193 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.241+0000 7fe7ef7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f0107c50 0x7fe7f019cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57776/0 (socket says 172.21.15.46:57776) 2024-04-26T21:25:36.193 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.241+0000 7fe7ef7fe700 1 -- 172.21.15.46:0/1927040270 learned_addr learned my addr 172.21.15.46:0/1927040270 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:36.194 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.241+0000 7fe7ef7fe700 1 -- 172.21.15.46:0/1927040270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7f019d1e0 con 0x7fe7f0107c50 2024-04-26T21:25:36.194 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.241+0000 7fe7ef7fe700 1 --2- 172.21.15.46:0/1927040270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f0107c50 0x7fe7f019cca0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe7e00119f0 tx=0x7fe7e0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:36.194 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.242+0000 7fe7ecff9700 1 -- 172.21.15.46:0/1927040270 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7e00039e0 con 0x7fe7f0107c50 2024-04-26T21:25:36.194 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.242+0000 7fe7ecff9700 1 -- 172.21.15.46:0/1927040270 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7e001b8f0 con 0x7fe7f0107c50 2024-04-26T21:25:36.194 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.242+0000 7fe7f5f08700 1 -- 172.21.15.46:0/1927040270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7f019d470 con 0x7fe7f0107c50 2024-04-26T21:25:36.194 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.242+0000 7fe7ecff9700 1 -- 172.21.15.46:0/1927040270 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7e00237a0 con 0x7fe7f0107c50 2024-04-26T21:25:36.194 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.242+0000 7fe7f5f08700 1 -- 172.21.15.46:0/1927040270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7f01a0160 con 0x7fe7f0107c50 2024-04-26T21:25:36.194 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.242+0000 7fe7ecff9700 1 -- 172.21.15.46:0/1927040270 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7fe7e0023aa0 con 0x7fe7f0107c50 2024-04-26T21:25:36.194 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.243+0000 7fe7ecff9700 1 --2- 172.21.15.46:0/1927040270 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe7d80395b0 0x7fe7d803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:36.194 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.243+0000 7fe7f5f08700 1 -- 172.21.15.46:0/1927040270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7f0064a80 con 0x7fe7f0107c50 2024-04-26T21:25:36.194 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.243+0000 7fe7ecff9700 1 -- 172.21.15.46:0/1927040270 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe7e004dfd0 con 0x7fe7f0107c50 2024-04-26T21:25:36.194 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.243+0000 7fe7eeffd700 1 --2- 172.21.15.46:0/1927040270 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe7d80395b0 0x7fe7d803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:36.194 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.243+0000 7fe7eeffd700 1 --2- 172.21.15.46:0/1927040270 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe7d80395b0 0x7fe7d803ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4006fd0 tx=0x7fe7e4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:36.195 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.248+0000 7fe7ecff9700 1 -- 172.21.15.46:0/1927040270 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe7e001f030 con 0x7fe7f0107c50 2024-04-26T21:25:36.195 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.459+0000 7fe7f5f08700 1 -- 172.21.15.46:0/1927040270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fe7f002e6e0 con 0x7fe7f0107c50 2024-04-26T21:25:36.195 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.459+0000 7fe7ecff9700 1 -- 172.21.15.46:0/1927040270 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7fe7e0018e30 con 0x7fe7f0107c50 2024-04-26T21:25:36.195 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.461+0000 7fe7f5f08700 1 -- 172.21.15.46:0/1927040270 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe7d80395b0 msgr2=0x7fe7d803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:36.195 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.461+0000 7fe7f5f08700 1 --2- 172.21.15.46:0/1927040270 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe7d80395b0 0x7fe7d803ba70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4006fd0 tx=0x7fe7e4006e40 comp rx=0 tx=0).stop 2024-04-26T21:25:36.195 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.461+0000 7fe7f5f08700 1 -- 172.21.15.46:0/1927040270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f0107c50 msgr2=0x7fe7f019cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:36.195 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.461+0000 7fe7f5f08700 1 --2- 172.21.15.46:0/1927040270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f0107c50 0x7fe7f019cca0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe7e00119f0 tx=0x7fe7e0011ad0 comp rx=0 tx=0).stop 2024-04-26T21:25:36.195 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.462+0000 7fe7f5f08700 1 -- 172.21.15.46:0/1927040270 shutdown_connections 2024-04-26T21:25:36.195 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.462+0000 7fe7f5f08700 1 --2- 172.21.15.46:0/1927040270 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe7d80395b0 0x7fe7d803ba70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:36.195 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.462+0000 7fe7f5f08700 1 --2- 172.21.15.46:0/1927040270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f0107c50 0x7fe7f019cca0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:36.195 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.462+0000 7fe7f5f08700 1 -- 172.21.15.46:0/1927040270 >> 172.21.15.46:0/1927040270 conn(0x7fe7f01031f0 msgr2=0x7fe7f0104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:36.195 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.462+0000 7fe7f5f08700 1 -- 172.21.15.46:0/1927040270 shutdown_connections 2024-04-26T21:25:36.195 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:35.462+0000 7fe7f5f08700 1 -- 172.21.15.46:0/1927040270 wait complete. 2024-04-26T21:25:36.200 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T21:25:36.200 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout disabled 2024-04-26T21:25:36.208 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T21:25:36.208 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2024-04-26T21:25:36.208 INFO:teuthology.orchestra.run.smithi046.stdout:firewalld.service is not enabled 2024-04-26T21:25:36.208 INFO:teuthology.orchestra.run.smithi046.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-26T21:25:36.210 INFO:teuthology.orchestra.run.smithi046.stdout:Ceph Dashboard is now available at: 2024-04-26T21:25:36.210 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:25:36.210 INFO:teuthology.orchestra.run.smithi046.stdout: URL: https://smithi046:8443/ 2024-04-26T21:25:36.210 INFO:teuthology.orchestra.run.smithi046.stdout: User: admin 2024-04-26T21:25:36.210 INFO:teuthology.orchestra.run.smithi046.stdout: Password: zgbw3umoru 2024-04-26T21:25:36.210 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:25:36.211 INFO:teuthology.orchestra.run.smithi046.stdout:Saving cluster configuration to /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/config directory 2024-04-26T21:25:36.211 INFO:teuthology.orchestra.run.smithi046.stdout:Enabling autotune for osd_memory_target 2024-04-26T21:25:37.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:37 smithi046 ceph-mon[29201]: mgrmap e13: x(active, since 6s) 2024-04-26T21:25:37.909 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.983+0000 7f09f7090700 1 Processor -- start 2024-04-26T21:25:37.909 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.983+0000 7f09f7090700 1 -- start start 2024-04-26T21:25:37.909 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.984+0000 7f09f7090700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09f0109840 0x7f09f010bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:37.909 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.984+0000 7f09f7090700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f09f007d6d0 con 0x7f09f0109840 2024-04-26T21:25:37.909 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.984+0000 7f09f4e2c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09f0109840 0x7f09f010bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:37.909 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.984+0000 7f09f4e2c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09f0109840 0x7f09f010bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48056/0 (socket says 172.21.15.46:48056) 2024-04-26T21:25:37.909 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.984+0000 7f09f4e2c700 1 -- 172.21.15.46:0/1692952689 learned_addr learned my addr 172.21.15.46:0/1692952689 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:37.909 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.984+0000 7f09f4e2c700 1 -- 172.21.15.46:0/1692952689 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f010c170 con 0x7f09f0109840 2024-04-26T21:25:37.909 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.984+0000 7f09f4e2c700 1 --2- 172.21.15.46:0/1692952689 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09f0109840 0x7f09f010bc30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f09e40098a0 tx=0x7f09e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28b1d746c4923ca server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:37.909 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.984+0000 7f09ef7fe700 1 -- 172.21.15.46:0/1692952689 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09e4004160 con 0x7f09f0109840 2024-04-26T21:25:37.909 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.985+0000 7f09ef7fe700 1 -- 172.21.15.46:0/1692952689 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09e4004300 con 0x7f09f0109840 2024-04-26T21:25:37.910 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.985+0000 7f09ef7fe700 1 -- 172.21.15.46:0/1692952689 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09e4011a20 con 0x7f09f0109840 2024-04-26T21:25:37.910 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.985+0000 7f09f7090700 1 -- 172.21.15.46:0/1692952689 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09f0109840 msgr2=0x7f09f010bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:37.910 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.985+0000 7f09f7090700 1 --2- 172.21.15.46:0/1692952689 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09f0109840 0x7f09f010bc30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f09e40098a0 tx=0x7f09e4009bb0 comp rx=0 tx=0).stop 2024-04-26T21:25:37.910 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.985+0000 7f09f7090700 1 -- 172.21.15.46:0/1692952689 shutdown_connections 2024-04-26T21:25:37.910 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.985+0000 7f09f7090700 1 --2- 172.21.15.46:0/1692952689 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09f0109840 0x7f09f010bc30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:37.910 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.985+0000 7f09f7090700 1 -- 172.21.15.46:0/1692952689 >> 172.21.15.46:0/1692952689 conn(0x7f09f01031d0 msgr2=0x7f09f0105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:37.910 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.985+0000 7f09f7090700 1 -- 172.21.15.46:0/1692952689 shutdown_connections 2024-04-26T21:25:37.910 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.985+0000 7f09f7090700 1 -- 172.21.15.46:0/1692952689 wait complete. 2024-04-26T21:25:37.910 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.986+0000 7f09f7090700 1 Processor -- start 2024-04-26T21:25:37.910 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.986+0000 7f09f7090700 1 -- start start 2024-04-26T21:25:37.910 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.986+0000 7f09f7090700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09f0109840 0x7f09f019cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:37.910 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.986+0000 7f09f7090700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f09f010d120 con 0x7f09f0109840 2024-04-26T21:25:37.910 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.986+0000 7f09f4e2c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09f0109840 0x7f09f019cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:37.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.986+0000 7f09f4e2c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09f0109840 0x7f09f019cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48064/0 (socket says 172.21.15.46:48064) 2024-04-26T21:25:37.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.986+0000 7f09f4e2c700 1 -- 172.21.15.46:0/975641292 learned_addr learned my addr 172.21.15.46:0/975641292 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:37.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.987+0000 7f09f4e2c700 1 -- 172.21.15.46:0/975641292 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f019d180 con 0x7f09f0109840 2024-04-26T21:25:37.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.987+0000 7f09f4e2c700 1 --2- 172.21.15.46:0/975641292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09f0109840 0x7f09f019cc40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f09e4004660 tx=0x7f09e4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:37.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.987+0000 7f09edffb700 1 -- 172.21.15.46:0/975641292 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09e4003aa0 con 0x7f09f0109840 2024-04-26T21:25:37.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.987+0000 7f09edffb700 1 -- 172.21.15.46:0/975641292 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09e401b8f0 con 0x7f09f0109840 2024-04-26T21:25:37.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.987+0000 7f09f7090700 1 -- 172.21.15.46:0/975641292 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09f019d410 con 0x7f09f0109840 2024-04-26T21:25:37.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.987+0000 7f09edffb700 1 -- 172.21.15.46:0/975641292 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09e4023900 con 0x7f09f0109840 2024-04-26T21:25:37.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.987+0000 7f09f7090700 1 -- 172.21.15.46:0/975641292 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09f01a0100 con 0x7f09f0109840 2024-04-26T21:25:37.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.988+0000 7f09edffb700 1 -- 172.21.15.46:0/975641292 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f09e4023bc0 con 0x7f09f0109840 2024-04-26T21:25:37.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.988+0000 7f09f7090700 1 -- 172.21.15.46:0/975641292 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09f0064a80 con 0x7f09f0109840 2024-04-26T21:25:37.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.988+0000 7f09edffb700 1 --2- 172.21.15.46:0/975641292 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f09e00396a0 0x7f09e003bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:37.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.988+0000 7f09edffb700 1 -- 172.21.15.46:0/975641292 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f09e404e1f0 con 0x7f09f0109840 2024-04-26T21:25:37.911 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.988+0000 7f09effff700 1 --2- 172.21.15.46:0/975641292 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f09e00396a0 0x7f09e003bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:37.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.989+0000 7f09effff700 1 --2- 172.21.15.46:0/975641292 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f09e00396a0 0x7f09e003bb60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f09dc006fd0 tx=0x7f09dc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:37.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:36.993+0000 7f09edffb700 1 -- 172.21.15.46:0/975641292 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f09e401f030 con 0x7f09f0109840 2024-04-26T21:25:37.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:37.201+0000 7f09f7090700 1 -- 172.21.15.46:0/975641292 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f09f01a0760 con 0x7f09f0109840 2024-04-26T21:25:37.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:37.201+0000 7f09edffb700 1 -- 172.21.15.46:0/975641292 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f09e401b440 con 0x7f09f0109840 2024-04-26T21:25:37.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:37.203+0000 7f09f7090700 1 -- 172.21.15.46:0/975641292 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f09e00396a0 msgr2=0x7f09e003bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:37.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:37.203+0000 7f09f7090700 1 --2- 172.21.15.46:0/975641292 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f09e00396a0 0x7f09e003bb60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f09dc006fd0 tx=0x7f09dc006e40 comp rx=0 tx=0).stop 2024-04-26T21:25:37.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:37.203+0000 7f09f7090700 1 -- 172.21.15.46:0/975641292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09f0109840 msgr2=0x7f09f019cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:37.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:37.203+0000 7f09f7090700 1 --2- 172.21.15.46:0/975641292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09f0109840 0x7f09f019cc40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f09e4004660 tx=0x7f09e4011aa0 comp rx=0 tx=0).stop 2024-04-26T21:25:37.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:37.203+0000 7f09f7090700 1 -- 172.21.15.46:0/975641292 shutdown_connections 2024-04-26T21:25:37.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:37.203+0000 7f09f7090700 1 --2- 172.21.15.46:0/975641292 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f09e00396a0 0x7f09e003bb60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:37.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:37.203+0000 7f09f7090700 1 --2- 172.21.15.46:0/975641292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09f0109840 0x7f09f019cc40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:37.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:37.203+0000 7f09f7090700 1 -- 172.21.15.46:0/975641292 >> 172.21.15.46:0/975641292 conn(0x7f09f01031d0 msgr2=0x7f09f0103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:37.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:37.204+0000 7f09f7090700 1 -- 172.21.15.46:0/975641292 shutdown_connections 2024-04-26T21:25:37.912 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:37.204+0000 7f09f7090700 1 -- 172.21.15.46:0/975641292 wait complete. 2024-04-26T21:25:39.206 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.307+0000 7f550cf05700 1 Processor -- start 2024-04-26T21:25:39.206 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.307+0000 7f550cf05700 1 -- start start 2024-04-26T21:25:39.206 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.307+0000 7f550cf05700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f550807f690 0x7f550807db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:39.206 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.307+0000 7f550cf05700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f550807e080 con 0x7f550807f690 2024-04-26T21:25:39.206 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.307+0000 7f550659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f550807f690 0x7f550807db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:39.206 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.307+0000 7f550659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f550807f690 0x7f550807db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48072/0 (socket says 172.21.15.46:48072) 2024-04-26T21:25:39.206 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.307+0000 7f550659c700 1 -- 172.21.15.46:0/2088543629 learned_addr learned my addr 172.21.15.46:0/2088543629 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:39.207 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.308+0000 7f550659c700 1 -- 172.21.15.46:0/2088543629 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f550807e200 con 0x7f550807f690 2024-04-26T21:25:39.207 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.308+0000 7f550659c700 1 --2- 172.21.15.46:0/2088543629 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f550807f690 0x7f550807db40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f54f80098a0 tx=0x7f54f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e15ddf40ae644373 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:39.207 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.308+0000 7f550559a700 1 -- 172.21.15.46:0/2088543629 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54f8004160 con 0x7f550807f690 2024-04-26T21:25:39.207 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.308+0000 7f550559a700 1 -- 172.21.15.46:0/2088543629 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54f8004300 con 0x7f550807f690 2024-04-26T21:25:39.207 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.308+0000 7f550559a700 1 -- 172.21.15.46:0/2088543629 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54f8011a20 con 0x7f550807f690 2024-04-26T21:25:39.207 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.309+0000 7f550cf05700 1 -- 172.21.15.46:0/2088543629 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f550807f690 msgr2=0x7f550807db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:39.207 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.309+0000 7f550cf05700 1 --2- 172.21.15.46:0/2088543629 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f550807f690 0x7f550807db40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f54f80098a0 tx=0x7f54f8009bb0 comp rx=0 tx=0).stop 2024-04-26T21:25:39.207 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.309+0000 7f550cf05700 1 -- 172.21.15.46:0/2088543629 shutdown_connections 2024-04-26T21:25:39.207 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.309+0000 7f550cf05700 1 --2- 172.21.15.46:0/2088543629 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f550807f690 0x7f550807db40 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:39.207 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.309+0000 7f550cf05700 1 -- 172.21.15.46:0/2088543629 >> 172.21.15.46:0/2088543629 conn(0x7f5508104370 msgr2=0x7f55081067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:39.207 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.309+0000 7f550cf05700 1 -- 172.21.15.46:0/2088543629 shutdown_connections 2024-04-26T21:25:39.207 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.309+0000 7f550cf05700 1 -- 172.21.15.46:0/2088543629 wait complete. 2024-04-26T21:25:39.208 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.310+0000 7f550cf05700 1 Processor -- start 2024-04-26T21:25:39.208 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.310+0000 7f550cf05700 1 -- start start 2024-04-26T21:25:39.208 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.310+0000 7f550cf05700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f550807f690 0x7f550819ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:39.208 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.310+0000 7f550cf05700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f550810f350 con 0x7f550807f690 2024-04-26T21:25:39.208 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.310+0000 7f550659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f550807f690 0x7f550819ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:39.208 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.310+0000 7f550659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f550807f690 0x7f550819ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48084/0 (socket says 172.21.15.46:48084) 2024-04-26T21:25:39.208 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.310+0000 7f550659c700 1 -- 172.21.15.46:0/2475781766 learned_addr learned my addr 172.21.15.46:0/2475781766 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:39.208 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.310+0000 7f550659c700 1 -- 172.21.15.46:0/2475781766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f550819f380 con 0x7f550807f690 2024-04-26T21:25:39.208 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.310+0000 7f550659c700 1 --2- 172.21.15.46:0/2475781766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f550807f690 0x7f550819ee40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f54f8004660 tx=0x7f54f8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:39.208 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.311+0000 7f54f77fe700 1 -- 172.21.15.46:0/2475781766 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54f8003aa0 con 0x7f550807f690 2024-04-26T21:25:39.208 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.311+0000 7f54f77fe700 1 -- 172.21.15.46:0/2475781766 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54f801b8f0 con 0x7f550807f690 2024-04-26T21:25:39.208 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.311+0000 7f550cf05700 1 -- 172.21.15.46:0/2475781766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f550819f610 con 0x7f550807f690 2024-04-26T21:25:39.208 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.311+0000 7f54f77fe700 1 -- 172.21.15.46:0/2475781766 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f54f8023900 con 0x7f550807f690 2024-04-26T21:25:39.208 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.311+0000 7f550cf05700 1 -- 172.21.15.46:0/2475781766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55081a2300 con 0x7f550807f690 2024-04-26T21:25:39.209 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.311+0000 7f54f77fe700 1 -- 172.21.15.46:0/2475781766 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f54f8023b70 con 0x7f550807f690 2024-04-26T21:25:39.209 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.312+0000 7f54f77fe700 1 --2- 172.21.15.46:0/2475781766 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f54f0039700 0x7f54f003bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:39.209 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.312+0000 7f54f77fe700 1 -- 172.21.15.46:0/2475781766 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f54f804e010 con 0x7f550807f690 2024-04-26T21:25:39.209 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.312+0000 7f5505d9b700 1 --2- 172.21.15.46:0/2475781766 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f54f0039700 0x7f54f003bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:39.209 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.312+0000 7f550cf05700 1 -- 172.21.15.46:0/2475781766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5508064a80 con 0x7f550807f690 2024-04-26T21:25:39.209 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.312+0000 7f5505d9b700 1 --2- 172.21.15.46:0/2475781766 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f54f0039700 0x7f54f003bbc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f54fc006fd0 tx=0x7f54fc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:39.209 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.317+0000 7f54f77fe700 1 -- 172.21.15.46:0/2475781766 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f54f801f030 con 0x7f550807f690 2024-04-26T21:25:39.209 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.643+0000 7f550cf05700 1 -- 172.21.15.46:0/2475781766 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f55081a24d0 con 0x7f550807f690 2024-04-26T21:25:39.209 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.646+0000 7f54f77fe700 1 -- 172.21.15.46:0/2475781766 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f54f8010380 con 0x7f550807f690 2024-04-26T21:25:39.209 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.647+0000 7f550cf05700 1 -- 172.21.15.46:0/2475781766 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f54f0039700 msgr2=0x7f54f003bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:39.209 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.647+0000 7f550cf05700 1 --2- 172.21.15.46:0/2475781766 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f54f0039700 0x7f54f003bbc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f54fc006fd0 tx=0x7f54fc006e40 comp rx=0 tx=0).stop 2024-04-26T21:25:39.209 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.648+0000 7f550cf05700 1 -- 172.21.15.46:0/2475781766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f550807f690 msgr2=0x7f550819ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:39.209 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.648+0000 7f550cf05700 1 --2- 172.21.15.46:0/2475781766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f550807f690 0x7f550819ee40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f54f8004660 tx=0x7f54f8011aa0 comp rx=0 tx=0).stop 2024-04-26T21:25:39.209 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.648+0000 7f550cf05700 1 -- 172.21.15.46:0/2475781766 shutdown_connections 2024-04-26T21:25:39.210 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.648+0000 7f550cf05700 1 --2- 172.21.15.46:0/2475781766 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f54f0039700 0x7f54f003bbc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:39.210 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.648+0000 7f550cf05700 1 --2- 172.21.15.46:0/2475781766 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f550807f690 0x7f550819ee40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:39.210 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.648+0000 7f550cf05700 1 -- 172.21.15.46:0/2475781766 >> 172.21.15.46:0/2475781766 conn(0x7f5508104370 msgr2=0x7f5508104e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:39.210 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.648+0000 7f550cf05700 1 -- 172.21.15.46:0/2475781766 shutdown_connections 2024-04-26T21:25:39.210 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-04-26T21:25:38.648+0000 7f550cf05700 1 -- 172.21.15.46:0/2475781766 wait complete. 2024-04-26T21:25:39.210 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-26T21:25:39.210 INFO:teuthology.orchestra.run.smithi046.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-26T21:25:39.210 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:25:39.210 INFO:teuthology.orchestra.run.smithi046.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-26T21:25:39.210 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:25:39.210 INFO:teuthology.orchestra.run.smithi046.stdout:Or, if you are only running a single cluster on this host: 2024-04-26T21:25:39.210 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:25:39.210 INFO:teuthology.orchestra.run.smithi046.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-26T21:25:39.210 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:25:39.211 INFO:teuthology.orchestra.run.smithi046.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-26T21:25:39.211 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:25:39.211 INFO:teuthology.orchestra.run.smithi046.stdout: ceph telemetry on 2024-04-26T21:25:39.211 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:25:39.211 INFO:teuthology.orchestra.run.smithi046.stdout:For more information see: 2024-04-26T21:25:39.211 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:25:39.211 INFO:teuthology.orchestra.run.smithi046.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-26T21:25:39.211 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:25:39.211 INFO:teuthology.orchestra.run.smithi046.stdout:Bootstrap complete. 2024-04-26T21:25:39.246 INFO:tasks.cephadm:Fetching config... 2024-04-26T21:25:39.246 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-26T21:25:39.247 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-26T21:25:39.263 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-26T21:25:39.263 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-26T21:25:39.263 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-26T21:25:39.319 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-26T21:25:39.320 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-26T21:25:39.320 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-26T21:25:39.384 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-26T21:25:39.385 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-26T21:25:39.385 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-26T21:25:39.441 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-26T21:25:39.441 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+sTE3O9pA8MQ+f+aP4yoPABL5ao5Aejrieq1oKNxiyF5H7Sjxh6lYDvd0gIdnSN73uPMMrJq1tBmlvswAku33BVejwwYDyCrlkRRwQG4IOfPg6mi8/nOtAXBz1Ur2yjtJ7IvB4C7LB5w6gwsMJcklqaBWvHboIeHWo6XAjJwNkrCA2DLIRsGMcDihYMMJiLuu+k/Fed1k9MdAKDxj9Uz7kcIz8DyA7WA2F7lSVaX4YKvo0vS8IcGEinmZ5Hih0E4V/q3JWSJ5ujDwYdKyElxeIeUhMpmrLX8fQ3tXHlJw3IJy/9cDYkrCDuKpUp9GYYQDbLuEGDzaLxrYCJMabgvVdqdH0td9yQvIiTjxcdO00N0LcemrfMTX9OkJvW9CVVl+KEWnJyL3T2bJ4d5Fgo4r8eA4WXgWgBNNwxTl8vIpMpLZCCaVzfIysKXF9k6H5mz8jBZ37JDMl0CNjQThqYe++e1MICbcV/yAx0CtEeSS1Ew9PXpf3FVccU7VOdvZ85s= ceph-36c7013a-0413-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T21:25:39.519 INFO:teuthology.orchestra.run.smithi046.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+sTE3O9pA8MQ+f+aP4yoPABL5ao5Aejrieq1oKNxiyF5H7Sjxh6lYDvd0gIdnSN73uPMMrJq1tBmlvswAku33BVejwwYDyCrlkRRwQG4IOfPg6mi8/nOtAXBz1Ur2yjtJ7IvB4C7LB5w6gwsMJcklqaBWvHboIeHWo6XAjJwNkrCA2DLIRsGMcDihYMMJiLuu+k/Fed1k9MdAKDxj9Uz7kcIz8DyA7WA2F7lSVaX4YKvo0vS8IcGEinmZ5Hih0E4V/q3JWSJ5ujDwYdKyElxeIeUhMpmrLX8fQ3tXHlJw3IJy/9cDYkrCDuKpUp9GYYQDbLuEGDzaLxrYCJMabgvVdqdH0td9yQvIiTjxcdO00N0LcemrfMTX9OkJvW9CVVl+KEWnJyL3T2bJ4d5Fgo4r8eA4WXgWgBNNwxTl8vIpMpLZCCaVzfIysKXF9k6H5mz8jBZ37JDMl0CNjQThqYe++e1MICbcV/yAx0CtEeSS1Ew9PXpf3FVccU7VOdvZ85s= ceph-36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:25:39.539 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+sTE3O9pA8MQ+f+aP4yoPABL5ao5Aejrieq1oKNxiyF5H7Sjxh6lYDvd0gIdnSN73uPMMrJq1tBmlvswAku33BVejwwYDyCrlkRRwQG4IOfPg6mi8/nOtAXBz1Ur2yjtJ7IvB4C7LB5w6gwsMJcklqaBWvHboIeHWo6XAjJwNkrCA2DLIRsGMcDihYMMJiLuu+k/Fed1k9MdAKDxj9Uz7kcIz8DyA7WA2F7lSVaX4YKvo0vS8IcGEinmZ5Hih0E4V/q3JWSJ5ujDwYdKyElxeIeUhMpmrLX8fQ3tXHlJw3IJy/9cDYkrCDuKpUp9GYYQDbLuEGDzaLxrYCJMabgvVdqdH0td9yQvIiTjxcdO00N0LcemrfMTX9OkJvW9CVVl+KEWnJyL3T2bJ4d5Fgo4r8eA4WXgWgBNNwxTl8vIpMpLZCCaVzfIysKXF9k6H5mz8jBZ37JDMl0CNjQThqYe++e1MICbcV/yAx0CtEeSS1Ew9PXpf3FVccU7VOdvZ85s= ceph-36c7013a-0413-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T21:25:39.579 INFO:teuthology.orchestra.run.smithi050.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+sTE3O9pA8MQ+f+aP4yoPABL5ao5Aejrieq1oKNxiyF5H7Sjxh6lYDvd0gIdnSN73uPMMrJq1tBmlvswAku33BVejwwYDyCrlkRRwQG4IOfPg6mi8/nOtAXBz1Ur2yjtJ7IvB4C7LB5w6gwsMJcklqaBWvHboIeHWo6XAjJwNkrCA2DLIRsGMcDihYMMJiLuu+k/Fed1k9MdAKDxj9Uz7kcIz8DyA7WA2F7lSVaX4YKvo0vS8IcGEinmZ5Hih0E4V/q3JWSJ5ujDwYdKyElxeIeUhMpmrLX8fQ3tXHlJw3IJy/9cDYkrCDuKpUp9GYYQDbLuEGDzaLxrYCJMabgvVdqdH0td9yQvIiTjxcdO00N0LcemrfMTX9OkJvW9CVVl+KEWnJyL3T2bJ4d5Fgo4r8eA4WXgWgBNNwxTl8vIpMpLZCCaVzfIysKXF9k6H5mz8jBZ37JDMl0CNjQThqYe++e1MICbcV/yAx0CtEeSS1Ew9PXpf3FVccU7VOdvZ85s= ceph-36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:25:39.591 DEBUG:teuthology.orchestra.run.smithi142:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+sTE3O9pA8MQ+f+aP4yoPABL5ao5Aejrieq1oKNxiyF5H7Sjxh6lYDvd0gIdnSN73uPMMrJq1tBmlvswAku33BVejwwYDyCrlkRRwQG4IOfPg6mi8/nOtAXBz1Ur2yjtJ7IvB4C7LB5w6gwsMJcklqaBWvHboIeHWo6XAjJwNkrCA2DLIRsGMcDihYMMJiLuu+k/Fed1k9MdAKDxj9Uz7kcIz8DyA7WA2F7lSVaX4YKvo0vS8IcGEinmZ5Hih0E4V/q3JWSJ5ujDwYdKyElxeIeUhMpmrLX8fQ3tXHlJw3IJy/9cDYkrCDuKpUp9GYYQDbLuEGDzaLxrYCJMabgvVdqdH0td9yQvIiTjxcdO00N0LcemrfMTX9OkJvW9CVVl+KEWnJyL3T2bJ4d5Fgo4r8eA4WXgWgBNNwxTl8vIpMpLZCCaVzfIysKXF9k6H5mz8jBZ37JDMl0CNjQThqYe++e1MICbcV/yAx0CtEeSS1Ew9PXpf3FVccU7VOdvZ85s= ceph-36c7013a-0413-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T21:25:39.630 INFO:teuthology.orchestra.run.smithi142.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+sTE3O9pA8MQ+f+aP4yoPABL5ao5Aejrieq1oKNxiyF5H7Sjxh6lYDvd0gIdnSN73uPMMrJq1tBmlvswAku33BVejwwYDyCrlkRRwQG4IOfPg6mi8/nOtAXBz1Ur2yjtJ7IvB4C7LB5w6gwsMJcklqaBWvHboIeHWo6XAjJwNkrCA2DLIRsGMcDihYMMJiLuu+k/Fed1k9MdAKDxj9Uz7kcIz8DyA7WA2F7lSVaX4YKvo0vS8IcGEinmZ5Hih0E4V/q3JWSJ5ujDwYdKyElxeIeUhMpmrLX8fQ3tXHlJw3IJy/9cDYkrCDuKpUp9GYYQDbLuEGDzaLxrYCJMabgvVdqdH0td9yQvIiTjxcdO00N0LcemrfMTX9OkJvW9CVVl+KEWnJyL3T2bJ4d5Fgo4r8eA4WXgWgBNNwxTl8vIpMpLZCCaVzfIysKXF9k6H5mz8jBZ37JDMl0CNjQThqYe++e1MICbcV/yAx0CtEeSS1Ew9PXpf3FVccU7VOdvZ85s= ceph-36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:25:39.641 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-26T21:25:39.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:39 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2475781766' entity='client.admin' 2024-04-26T21:25:39.885 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:25:42.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.171+0000 7f59648b6700 1 -- 172.21.15.46:0/3835166281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5960103930 msgr2=0x7f5960105d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:42.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.171+0000 7f59648b6700 1 --2- 172.21.15.46:0/3835166281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5960103930 0x7f5960105d20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5950009910 tx=0x7f5950009c20 comp rx=0 tx=0).stop 2024-04-26T21:25:42.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.171+0000 7f59648b6700 1 -- 172.21.15.46:0/3835166281 shutdown_connections 2024-04-26T21:25:42.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.171+0000 7f59648b6700 1 --2- 172.21.15.46:0/3835166281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5960103930 0x7f5960105d20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:42.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.171+0000 7f59648b6700 1 -- 172.21.15.46:0/3835166281 >> 172.21.15.46:0/3835166281 conn(0x7f59600fd360 msgr2=0x7f59600ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:42.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.171+0000 7f59648b6700 1 -- 172.21.15.46:0/3835166281 shutdown_connections 2024-04-26T21:25:42.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.171+0000 7f59648b6700 1 -- 172.21.15.46:0/3835166281 wait complete. 2024-04-26T21:25:42.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.172+0000 7f59648b6700 1 Processor -- start 2024-04-26T21:25:42.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.172+0000 7f59648b6700 1 -- start start 2024-04-26T21:25:42.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.172+0000 7f59648b6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5960103930 0x7f59601031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:42.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.172+0000 7f59648b6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5960108d80 con 0x7f5960103930 2024-04-26T21:25:42.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.172+0000 7f595e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5960103930 0x7f59601031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:42.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.172+0000 7f595e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5960103930 0x7f59601031b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48108/0 (socket says 172.21.15.46:48108) 2024-04-26T21:25:42.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.172+0000 7f595e59c700 1 -- 172.21.15.46:0/342988311 learned_addr learned my addr 172.21.15.46:0/342988311 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:42.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.172+0000 7f595e59c700 1 -- 172.21.15.46:0/342988311 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59601036f0 con 0x7f5960103930 2024-04-26T21:25:42.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.173+0000 7f595e59c700 1 --2- 172.21.15.46:0/342988311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5960103930 0x7f59601031b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5950005b40 tx=0x7f5950004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:42.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.173+0000 7f594f7fe700 1 -- 172.21.15.46:0/342988311 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5950027070 con 0x7f5960103930 2024-04-26T21:25:42.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.173+0000 7f594f7fe700 1 -- 172.21.15.46:0/342988311 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f595000be10 con 0x7f5960103930 2024-04-26T21:25:42.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.173+0000 7f59648b6700 1 -- 172.21.15.46:0/342988311 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59601018e0 con 0x7f5960103930 2024-04-26T21:25:42.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.173+0000 7f594f7fe700 1 -- 172.21.15.46:0/342988311 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5950022d80 con 0x7f5960103930 2024-04-26T21:25:42.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.173+0000 7f59648b6700 1 -- 172.21.15.46:0/342988311 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5960101e20 con 0x7f5960103930 2024-04-26T21:25:42.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.174+0000 7f594f7fe700 1 -- 172.21.15.46:0/342988311 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f595002a070 con 0x7f5960103930 2024-04-26T21:25:42.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.174+0000 7f594f7fe700 1 --2- 172.21.15.46:0/342988311 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5948039650 0x7f594803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:42.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.174+0000 7f594f7fe700 1 -- 172.21.15.46:0/342988311 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f595001e030 con 0x7f5960103930 2024-04-26T21:25:42.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.174+0000 7f59648b6700 1 -- 172.21.15.46:0/342988311 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5960109500 con 0x7f5960103930 2024-04-26T21:25:42.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.174+0000 7f595dd9b700 1 --2- 172.21.15.46:0/342988311 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5948039650 0x7f594803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:42.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.174+0000 7f595dd9b700 1 --2- 172.21.15.46:0/342988311 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5948039650 0x7f594803bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5954006fd0 tx=0x7f5954006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:42.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.179+0000 7f594f7fe700 1 -- 172.21.15.46:0/342988311 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5950013070 con 0x7f5960103930 2024-04-26T21:25:42.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.389+0000 7f59648b6700 1 -- 172.21.15.46:0/342988311 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f59601917a0 con 0x7f5960103930 2024-04-26T21:25:42.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.392+0000 7f594f7fe700 1 -- 172.21.15.46:0/342988311 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f5950035030 con 0x7f5960103930 2024-04-26T21:25:42.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.395+0000 7f59648b6700 1 -- 172.21.15.46:0/342988311 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5948039650 msgr2=0x7f594803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:42.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.395+0000 7f59648b6700 1 --2- 172.21.15.46:0/342988311 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5948039650 0x7f594803bb10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5954006fd0 tx=0x7f5954006e40 comp rx=0 tx=0).stop 2024-04-26T21:25:42.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.395+0000 7f59648b6700 1 -- 172.21.15.46:0/342988311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5960103930 msgr2=0x7f59601031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:42.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.395+0000 7f59648b6700 1 --2- 172.21.15.46:0/342988311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5960103930 0x7f59601031b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5950005b40 tx=0x7f5950004f80 comp rx=0 tx=0).stop 2024-04-26T21:25:42.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.395+0000 7f59648b6700 1 -- 172.21.15.46:0/342988311 shutdown_connections 2024-04-26T21:25:42.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.395+0000 7f59648b6700 1 --2- 172.21.15.46:0/342988311 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5948039650 0x7f594803bb10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:42.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.395+0000 7f59648b6700 1 --2- 172.21.15.46:0/342988311 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5960103930 0x7f59601031b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:42.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.395+0000 7f59648b6700 1 -- 172.21.15.46:0/342988311 >> 172.21.15.46:0/342988311 conn(0x7f59600fd360 msgr2=0x7f59600fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:42.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.395+0000 7f59648b6700 1 -- 172.21.15.46:0/342988311 shutdown_connections 2024-04-26T21:25:42.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:42.395+0000 7f59648b6700 1 -- 172.21.15.46:0/342988311 wait complete. 2024-04-26T21:25:43.219 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-26T21:25:43.220 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-26T21:25:43.526 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:43 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/342988311' entity='client.admin' 2024-04-26T21:25:43.892 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:25:45.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:44 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:25:45.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:44 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:25:45.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:44 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi046", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:25:45.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:44 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:25:45.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:44 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:45.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:44 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:25:45.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.312+0000 7f39228f1700 1 -- 172.21.15.46:0/3084319581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f391c1044d0 msgr2=0x7f391c1048f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:45.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.312+0000 7f39228f1700 1 --2- 172.21.15.46:0/3084319581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f391c1044d0 0x7f391c1048f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f390c009910 tx=0x7f390c009c20 comp rx=0 tx=0).stop 2024-04-26T21:25:45.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.313+0000 7f39228f1700 1 -- 172.21.15.46:0/3084319581 shutdown_connections 2024-04-26T21:25:45.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.313+0000 7f39228f1700 1 --2- 172.21.15.46:0/3084319581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f391c1044d0 0x7f391c1048f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:45.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.313+0000 7f39228f1700 1 -- 172.21.15.46:0/3084319581 >> 172.21.15.46:0/3084319581 conn(0x7f391c0fe450 msgr2=0x7f391c100870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:45.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.313+0000 7f39228f1700 1 -- 172.21.15.46:0/3084319581 shutdown_connections 2024-04-26T21:25:45.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.313+0000 7f39228f1700 1 -- 172.21.15.46:0/3084319581 wait complete. 2024-04-26T21:25:45.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.313+0000 7f39228f1700 1 Processor -- start 2024-04-26T21:25:45.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.313+0000 7f39228f1700 1 -- start start 2024-04-26T21:25:45.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.314+0000 7f39228f1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f391c1044d0 0x7f391c198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:45.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.314+0000 7f39228f1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f391c108e50 con 0x7f391c1044d0 2024-04-26T21:25:45.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.314+0000 7f391bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f391c1044d0 0x7f391c198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:45.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.314+0000 7f391bfff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f391c1044d0 0x7f391c198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48116/0 (socket says 172.21.15.46:48116) 2024-04-26T21:25:45.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.314+0000 7f391bfff700 1 -- 172.21.15.46:0/3049158458 learned_addr learned my addr 172.21.15.46:0/3049158458 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:45.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.314+0000 7f391bfff700 1 -- 172.21.15.46:0/3049158458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f391c198ad0 con 0x7f391c1044d0 2024-04-26T21:25:45.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.314+0000 7f391bfff700 1 --2- 172.21.15.46:0/3049158458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f391c1044d0 0x7f391c198590 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f390c005b40 tx=0x7f390c004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:45.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.315+0000 7f39197fa700 1 -- 172.21.15.46:0/3049158458 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f390c027070 con 0x7f391c1044d0 2024-04-26T21:25:45.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.315+0000 7f39197fa700 1 -- 172.21.15.46:0/3049158458 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f390c00be10 con 0x7f391c1044d0 2024-04-26T21:25:45.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.315+0000 7f39228f1700 1 -- 172.21.15.46:0/3049158458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f391c198d60 con 0x7f391c1044d0 2024-04-26T21:25:45.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.315+0000 7f39197fa700 1 -- 172.21.15.46:0/3049158458 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f390c022de0 con 0x7f391c1044d0 2024-04-26T21:25:45.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.315+0000 7f39228f1700 1 -- 172.21.15.46:0/3049158458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f391c19ba50 con 0x7f391c1044d0 2024-04-26T21:25:45.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.315+0000 7f39197fa700 1 -- 172.21.15.46:0/3049158458 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f390c02a070 con 0x7f391c1044d0 2024-04-26T21:25:45.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.316+0000 7f39197fa700 1 --2- 172.21.15.46:0/3049158458 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f39040396b0 0x7f390403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:45.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.316+0000 7f39228f1700 1 -- 172.21.15.46:0/3049158458 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f391c1095d0 con 0x7f391c1044d0 2024-04-26T21:25:45.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.316+0000 7f39197fa700 1 -- 172.21.15.46:0/3049158458 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f390c01e030 con 0x7f391c1044d0 2024-04-26T21:25:45.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.316+0000 7f391b7fe700 1 --2- 172.21.15.46:0/3049158458 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f39040396b0 0x7f390403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:45.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.316+0000 7f391b7fe700 1 --2- 172.21.15.46:0/3049158458 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f39040396b0 0x7f390403bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3910006fd0 tx=0x7f3910006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:45.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.321+0000 7f39197fa700 1 -- 172.21.15.46:0/3049158458 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f390c013070 con 0x7f391c1044d0 2024-04-26T21:25:45.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.532+0000 7f39228f1700 1 -- 172.21.15.46:0/3049158458 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f391c063860 con 0x7f39040396b0 2024-04-26T21:25:45.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.536+0000 7f39197fa700 1 -- 172.21.15.46:0/3049158458 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f391c063860 con 0x7f39040396b0 2024-04-26T21:25:45.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.538+0000 7f39228f1700 1 -- 172.21.15.46:0/3049158458 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f39040396b0 msgr2=0x7f390403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:45.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.538+0000 7f39228f1700 1 --2- 172.21.15.46:0/3049158458 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f39040396b0 0x7f390403bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3910006fd0 tx=0x7f3910006e40 comp rx=0 tx=0).stop 2024-04-26T21:25:45.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.538+0000 7f39228f1700 1 -- 172.21.15.46:0/3049158458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f391c1044d0 msgr2=0x7f391c198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:45.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.538+0000 7f39228f1700 1 --2- 172.21.15.46:0/3049158458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f391c1044d0 0x7f391c198590 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f390c005b40 tx=0x7f390c004f80 comp rx=0 tx=0).stop 2024-04-26T21:25:45.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.538+0000 7f39228f1700 1 -- 172.21.15.46:0/3049158458 shutdown_connections 2024-04-26T21:25:45.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.538+0000 7f39228f1700 1 --2- 172.21.15.46:0/3049158458 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f39040396b0 0x7f390403bb70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:45.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.538+0000 7f39228f1700 1 --2- 172.21.15.46:0/3049158458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f391c1044d0 0x7f391c198590 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:45.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.538+0000 7f39228f1700 1 -- 172.21.15.46:0/3049158458 >> 172.21.15.46:0/3049158458 conn(0x7f391c0fe450 msgr2=0x7f391c0fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:45.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.538+0000 7f39228f1700 1 -- 172.21.15.46:0/3049158458 shutdown_connections 2024-04-26T21:25:45.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:45.538+0000 7f39228f1700 1 -- 172.21.15.46:0/3049158458 wait complete. 2024-04-26T21:25:46.213 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi050 2024-04-26T21:25:46.213 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T21:25:46.214 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/etc/ceph/ceph.conf 2024-04-26T21:25:46.234 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T21:25:46.234 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:25:46.293 INFO:tasks.cephadm:Adding host smithi050 to orchestrator... 2024-04-26T21:25:46.293 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph orch host add smithi050 2024-04-26T21:25:46.561 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:25:46.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:46 smithi046 ceph-mon[29201]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:25:46.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:46 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:25:46.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:46 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:46.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:46 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:46.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:46 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:25:46.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:46 smithi046 ceph-mon[29201]: Updating smithi046:/etc/ceph/ceph.conf 2024-04-26T21:25:46.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:46 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:25:46.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:46 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:25:46.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:46 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:25:47.733 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:47 smithi046 ceph-mon[29201]: Updating smithi046:/var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:25:47.734 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:47 smithi046 ceph-mon[29201]: Updating smithi046:/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:25:47.734 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:47 smithi046 ceph-mon[29201]: Updating smithi046:/var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T21:25:47.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.984+0000 7f4f34888700 1 -- 172.21.15.46:0/1504830188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f30101f90 msgr2=0x7f4f30102370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:47.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.984+0000 7f4f34888700 1 --2- 172.21.15.46:0/1504830188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f30101f90 0x7f4f30102370 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4f20009910 tx=0x7f4f20009c20 comp rx=0 tx=0).stop 2024-04-26T21:25:47.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.984+0000 7f4f34888700 1 -- 172.21.15.46:0/1504830188 shutdown_connections 2024-04-26T21:25:47.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.984+0000 7f4f34888700 1 --2- 172.21.15.46:0/1504830188 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f30101f90 0x7f4f30102370 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:47.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.984+0000 7f4f34888700 1 -- 172.21.15.46:0/1504830188 >> 172.21.15.46:0/1504830188 conn(0x7f4f300fd820 msgr2=0x7f4f300ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:47.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.984+0000 7f4f34888700 1 -- 172.21.15.46:0/1504830188 shutdown_connections 2024-04-26T21:25:47.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.984+0000 7f4f34888700 1 -- 172.21.15.46:0/1504830188 wait complete. 2024-04-26T21:25:47.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.985+0000 7f4f34888700 1 Processor -- start 2024-04-26T21:25:47.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.985+0000 7f4f34888700 1 -- start start 2024-04-26T21:25:47.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.985+0000 7f4f34888700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f30101f90 0x7f4f3019e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:47.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.986+0000 7f4f34888700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3010c210 con 0x7f4f30101f90 2024-04-26T21:25:47.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.986+0000 7f4f2e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f30101f90 0x7f4f3019e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:47.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.986+0000 7f4f2e59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f30101f90 0x7f4f3019e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60250/0 (socket says 172.21.15.46:60250) 2024-04-26T21:25:47.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.986+0000 7f4f2e59c700 1 -- 172.21.15.46:0/3213707574 learned_addr learned my addr 172.21.15.46:0/3213707574 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:47.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.986+0000 7f4f2e59c700 1 -- 172.21.15.46:0/3213707574 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f3019eb20 con 0x7f4f30101f90 2024-04-26T21:25:47.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.986+0000 7f4f2e59c700 1 --2- 172.21.15.46:0/3213707574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f30101f90 0x7f4f3019e5e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4f20006010 tx=0x7f4f20005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:47.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.986+0000 7f4f277fe700 1 -- 172.21.15.46:0/3213707574 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f20026070 con 0x7f4f30101f90 2024-04-26T21:25:47.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.986+0000 7f4f34888700 1 -- 172.21.15.46:0/3213707574 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f3019ee10 con 0x7f4f30101f90 2024-04-26T21:25:47.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.986+0000 7f4f277fe700 1 -- 172.21.15.46:0/3213707574 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f2000bba0 con 0x7f4f30101f90 2024-04-26T21:25:47.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.986+0000 7f4f277fe700 1 -- 172.21.15.46:0/3213707574 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4f20003910 con 0x7f4f30101f90 2024-04-26T21:25:47.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.986+0000 7f4f34888700 1 -- 172.21.15.46:0/3213707574 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f30198880 con 0x7f4f30101f90 2024-04-26T21:25:47.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.987+0000 7f4f277fe700 1 -- 172.21.15.46:0/3213707574 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f4f20029070 con 0x7f4f30101f90 2024-04-26T21:25:47.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.987+0000 7f4f34888700 1 -- 172.21.15.46:0/3213707574 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f10005330 con 0x7f4f30101f90 2024-04-26T21:25:47.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.988+0000 7f4f277fe700 1 --2- 172.21.15.46:0/3213707574 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4f1c039650 0x7f4f1c03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:47.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.988+0000 7f4f277fe700 1 -- 172.21.15.46:0/3213707574 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4f2001e070 con 0x7f4f30101f90 2024-04-26T21:25:47.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.988+0000 7f4f2dd9b700 1 --2- 172.21.15.46:0/3213707574 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4f1c039650 0x7f4f1c03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:47.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.988+0000 7f4f2dd9b700 1 --2- 172.21.15.46:0/3213707574 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4f1c039650 0x7f4f1c03bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4f18006fd0 tx=0x7f4f18006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:47.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:47.993+0000 7f4f277fe700 1 -- 172.21.15.46:0/3213707574 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4f20013070 con 0x7f4f30101f90 2024-04-26T21:25:48.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:48.204+0000 7f4f34888700 1 -- 172.21.15.46:0/3213707574 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi050", "target": ["mon-mgr", ""]}) -- 0x7f4f10000bc0 con 0x7f4f1c039650 2024-04-26T21:25:49.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:48 smithi046 ceph-mon[29201]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi050", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:25:50.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:49 smithi046 ceph-mon[29201]: Deploying cephadm binary to smithi050 2024-04-26T21:25:51.204 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:50 smithi046 ceph-mon[29201]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:25:52.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:52.507+0000 7f4f277fe700 1 -- 172.21.15.46:0/3213707574 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f4f10000bc0 con 0x7f4f1c039650 2024-04-26T21:25:52.509 INFO:teuthology.orchestra.run.smithi046.stdout:Added host 'smithi050' with addr '172.21.15.50' 2024-04-26T21:25:52.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:52.510+0000 7f4f34888700 1 -- 172.21.15.46:0/3213707574 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4f1c039650 msgr2=0x7f4f1c03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:52.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:52.510+0000 7f4f34888700 1 --2- 172.21.15.46:0/3213707574 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4f1c039650 0x7f4f1c03bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4f18006fd0 tx=0x7f4f18006e40 comp rx=0 tx=0).stop 2024-04-26T21:25:52.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:52.510+0000 7f4f34888700 1 -- 172.21.15.46:0/3213707574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f30101f90 msgr2=0x7f4f3019e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:52.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:52.510+0000 7f4f34888700 1 --2- 172.21.15.46:0/3213707574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f30101f90 0x7f4f3019e5e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4f20006010 tx=0x7f4f20005780 comp rx=0 tx=0).stop 2024-04-26T21:25:52.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:52.510+0000 7f4f34888700 1 -- 172.21.15.46:0/3213707574 shutdown_connections 2024-04-26T21:25:52.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:52.510+0000 7f4f34888700 1 --2- 172.21.15.46:0/3213707574 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4f1c039650 0x7f4f1c03bb10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:52.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:52.510+0000 7f4f34888700 1 --2- 172.21.15.46:0/3213707574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f30101f90 0x7f4f3019e5e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:52.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:52.510+0000 7f4f34888700 1 -- 172.21.15.46:0/3213707574 >> 172.21.15.46:0/3213707574 conn(0x7f4f300fd820 msgr2=0x7f4f3006b290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:52.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:52.510+0000 7f4f34888700 1 -- 172.21.15.46:0/3213707574 shutdown_connections 2024-04-26T21:25:52.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:52.510+0000 7f4f34888700 1 -- 172.21.15.46:0/3213707574 wait complete. 2024-04-26T21:25:53.239 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-26T21:25:53.488 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:25:53.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:53 smithi046 ceph-mon[29201]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:25:53.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:53 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:25:53.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:53 smithi046 ceph-mon[29201]: Added host smithi050 2024-04-26T21:25:53.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:53 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:53.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:53 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:25:54.729 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:54 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:25:54.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.982+0000 7face34a9700 1 -- 172.21.15.46:0/2656001453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facdc103090 msgr2=0x7facdc103470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:54.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.983+0000 7face34a9700 1 --2- 172.21.15.46:0/2656001453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facdc103090 0x7facdc103470 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7facd8009910 tx=0x7facd8009c20 comp rx=0 tx=0).stop 2024-04-26T21:25:54.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.983+0000 7face34a9700 1 -- 172.21.15.46:0/2656001453 shutdown_connections 2024-04-26T21:25:54.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.983+0000 7face34a9700 1 --2- 172.21.15.46:0/2656001453 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facdc103090 0x7facdc103470 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:54.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.983+0000 7face34a9700 1 -- 172.21.15.46:0/2656001453 >> 172.21.15.46:0/2656001453 conn(0x7facdc0fe960 msgr2=0x7facdc100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:54.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.983+0000 7face34a9700 1 -- 172.21.15.46:0/2656001453 shutdown_connections 2024-04-26T21:25:54.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.983+0000 7face34a9700 1 -- 172.21.15.46:0/2656001453 wait complete. 2024-04-26T21:25:54.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.983+0000 7face34a9700 1 Processor -- start 2024-04-26T21:25:54.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.983+0000 7face34a9700 1 -- start start 2024-04-26T21:25:54.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.984+0000 7face34a9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facdc103090 0x7facdc19e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:54.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.984+0000 7face34a9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7facdc10d310 con 0x7facdc103090 2024-04-26T21:25:54.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.984+0000 7face1245700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facdc103090 0x7facdc19e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:54.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.984+0000 7face1245700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facdc103090 0x7facdc19e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60268/0 (socket says 172.21.15.46:60268) 2024-04-26T21:25:54.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.984+0000 7face1245700 1 -- 172.21.15.46:0/190026926 learned_addr learned my addr 172.21.15.46:0/190026926 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:54.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.984+0000 7face1245700 1 -- 172.21.15.46:0/190026926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facdc19eaf0 con 0x7facdc103090 2024-04-26T21:25:54.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.984+0000 7face1245700 1 --2- 172.21.15.46:0/190026926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facdc103090 0x7facdc19e5b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7facd8006010 tx=0x7facd8005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:54.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.984+0000 7facd27fc700 1 -- 172.21.15.46:0/190026926 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7facd8026070 con 0x7facdc103090 2024-04-26T21:25:54.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.984+0000 7facd27fc700 1 -- 172.21.15.46:0/190026926 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7facd800bba0 con 0x7facdc103090 2024-04-26T21:25:54.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.984+0000 7face34a9700 1 -- 172.21.15.46:0/190026926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facdc19ede0 con 0x7facdc103090 2024-04-26T21:25:54.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.984+0000 7facd27fc700 1 -- 172.21.15.46:0/190026926 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7facd8003910 con 0x7facdc103090 2024-04-26T21:25:54.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.984+0000 7face34a9700 1 -- 172.21.15.46:0/190026926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facdc198850 con 0x7facdc103090 2024-04-26T21:25:54.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.985+0000 7facd27fc700 1 -- 172.21.15.46:0/190026926 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7facd8029070 con 0x7facdc103090 2024-04-26T21:25:54.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.985+0000 7facd27fc700 1 --2- 172.21.15.46:0/190026926 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7facc8039650 0x7facc803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:54.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.985+0000 7facd27fc700 1 -- 172.21.15.46:0/190026926 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7facd801e070 con 0x7facdc103090 2024-04-26T21:25:54.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.986+0000 7face0a44700 1 --2- 172.21.15.46:0/190026926 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7facc8039650 0x7facc803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:54.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.986+0000 7face34a9700 1 -- 172.21.15.46:0/190026926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facdc104570 con 0x7facdc103090 2024-04-26T21:25:54.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.986+0000 7face0a44700 1 --2- 172.21.15.46:0/190026926 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7facc8039650 0x7facc803bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7faccc006fd0 tx=0x7faccc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:54.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:54.991+0000 7facd27fc700 1 -- 172.21.15.46:0/190026926 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7facd8030070 con 0x7facdc103090 2024-04-26T21:25:55.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:55.207+0000 7face34a9700 1 -- 172.21.15.46:0/190026926 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7facdc19efb0 con 0x7facc8039650 2024-04-26T21:25:55.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:55.209+0000 7facd27fc700 1 -- 172.21.15.46:0/190026926 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7facdc19efb0 con 0x7facc8039650 2024-04-26T21:25:55.210 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:25:55.210 INFO:teuthology.orchestra.run.smithi046.stdout:[{"addr": "172.21.15.46", "hostname": "smithi046", "labels": [], "status": ""}, {"addr": "172.21.15.50", "hostname": "smithi050", "labels": [], "status": ""}] 2024-04-26T21:25:55.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:55.212+0000 7face34a9700 1 -- 172.21.15.46:0/190026926 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7facc8039650 msgr2=0x7facc803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:55.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:55.212+0000 7face34a9700 1 --2- 172.21.15.46:0/190026926 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7facc8039650 0x7facc803bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7faccc006fd0 tx=0x7faccc006e40 comp rx=0 tx=0).stop 2024-04-26T21:25:55.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:55.212+0000 7face34a9700 1 -- 172.21.15.46:0/190026926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facdc103090 msgr2=0x7facdc19e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:55.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:55.212+0000 7face34a9700 1 --2- 172.21.15.46:0/190026926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facdc103090 0x7facdc19e5b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7facd8006010 tx=0x7facd8005780 comp rx=0 tx=0).stop 2024-04-26T21:25:55.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:55.212+0000 7face34a9700 1 -- 172.21.15.46:0/190026926 shutdown_connections 2024-04-26T21:25:55.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:55.212+0000 7face34a9700 1 --2- 172.21.15.46:0/190026926 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7facc8039650 0x7facc803bb10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:55.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:55.212+0000 7face34a9700 1 --2- 172.21.15.46:0/190026926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facdc103090 0x7facdc19e5b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:55.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:55.212+0000 7face34a9700 1 -- 172.21.15.46:0/190026926 >> 172.21.15.46:0/190026926 conn(0x7facdc0fe960 msgr2=0x7facdc104a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:55.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:55.212+0000 7face34a9700 1 -- 172.21.15.46:0/190026926 shutdown_connections 2024-04-26T21:25:55.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:55.213+0000 7face34a9700 1 -- 172.21.15.46:0/190026926 wait complete. 2024-04-26T21:25:55.765 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:55 smithi046 ceph-mon[29201]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:25:55.765 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:55 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:25:55.795 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi142 2024-04-26T21:25:55.795 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-04-26T21:25:55.795 DEBUG:teuthology.orchestra.run.smithi142:> dd of=/etc/ceph/ceph.conf 2024-04-26T21:25:55.814 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-04-26T21:25:55.814 DEBUG:teuthology.orchestra.run.smithi142:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:25:55.871 INFO:tasks.cephadm:Adding host smithi142 to orchestrator... 2024-04-26T21:25:55.872 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph orch host add smithi142 2024-04-26T21:25:56.115 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:25:56.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:56 smithi046 ceph-mon[29201]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:25:57.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.632+0000 7f4eaeb89700 1 -- 172.21.15.46:0/1785964644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ea8103090 msgr2=0x7f4ea8103470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:57.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.632+0000 7f4eaeb89700 1 --2- 172.21.15.46:0/1785964644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ea8103090 0x7f4ea8103470 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c009910 tx=0x7f4e9c009c20 comp rx=0 tx=0).stop 2024-04-26T21:25:57.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.632+0000 7f4eaeb89700 1 -- 172.21.15.46:0/1785964644 shutdown_connections 2024-04-26T21:25:57.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.632+0000 7f4eaeb89700 1 --2- 172.21.15.46:0/1785964644 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ea8103090 0x7f4ea8103470 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:57.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.632+0000 7f4eaeb89700 1 -- 172.21.15.46:0/1785964644 >> 172.21.15.46:0/1785964644 conn(0x7f4ea80fe940 msgr2=0x7f4ea8100d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:57.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.632+0000 7f4eaeb89700 1 -- 172.21.15.46:0/1785964644 shutdown_connections 2024-04-26T21:25:57.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.632+0000 7f4eaeb89700 1 -- 172.21.15.46:0/1785964644 wait complete. 2024-04-26T21:25:57.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.632+0000 7f4eaeb89700 1 Processor -- start 2024-04-26T21:25:57.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.633+0000 7f4eaeb89700 1 -- start start 2024-04-26T21:25:57.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.633+0000 7f4eaeb89700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ea8103090 0x7f4ea819e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:57.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.633+0000 7f4eaeb89700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea810d310 con 0x7f4ea8103090 2024-04-26T21:25:57.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.633+0000 7f4eac925700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ea8103090 0x7f4ea819e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:57.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.633+0000 7f4eac925700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ea8103090 0x7f4ea819e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53142/0 (socket says 172.21.15.46:53142) 2024-04-26T21:25:57.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.633+0000 7f4eac925700 1 -- 172.21.15.46:0/1460467787 learned_addr learned my addr 172.21.15.46:0/1460467787 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:25:57.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.633+0000 7f4eac925700 1 -- 172.21.15.46:0/1460467787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ea819eb00 con 0x7f4ea8103090 2024-04-26T21:25:57.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.633+0000 7f4eac925700 1 --2- 172.21.15.46:0/1460467787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ea8103090 0x7f4ea819e5c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c0056d0 tx=0x7f4e9c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:57.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.633+0000 7f4ea5ffb700 1 -- 172.21.15.46:0/1460467787 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e9c026070 con 0x7f4ea8103090 2024-04-26T21:25:57.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.633+0000 7f4ea5ffb700 1 -- 172.21.15.46:0/1460467787 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e9c00bba0 con 0x7f4ea8103090 2024-04-26T21:25:57.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.633+0000 7f4eaeb89700 1 -- 172.21.15.46:0/1460467787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ea819edf0 con 0x7f4ea8103090 2024-04-26T21:25:57.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.634+0000 7f4ea5ffb700 1 -- 172.21.15.46:0/1460467787 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e9c003920 con 0x7f4ea8103090 2024-04-26T21:25:57.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.634+0000 7f4eaeb89700 1 -- 172.21.15.46:0/1460467787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ea8198860 con 0x7f4ea8103090 2024-04-26T21:25:57.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.635+0000 7f4ea5ffb700 1 -- 172.21.15.46:0/1460467787 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f4e9c029070 con 0x7f4ea8103090 2024-04-26T21:25:57.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.635+0000 7f4eaeb89700 1 -- 172.21.15.46:0/1460467787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ea8104570 con 0x7f4ea8103090 2024-04-26T21:25:57.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.635+0000 7f4ea5ffb700 1 --2- 172.21.15.46:0/1460467787 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4e980396b0 0x7f4e9803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:57.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.635+0000 7f4ea5ffb700 1 -- 172.21.15.46:0/1460467787 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4e9c01e070 con 0x7f4ea8103090 2024-04-26T21:25:57.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.635+0000 7f4ea7fff700 1 --2- 172.21.15.46:0/1460467787 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4e980396b0 0x7f4e9803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:57.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.636+0000 7f4ea7fff700 1 --2- 172.21.15.46:0/1460467787 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4e980396b0 0x7f4e9803bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4e94006fd0 tx=0x7f4e94006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:57.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.640+0000 7f4ea5ffb700 1 -- 172.21.15.46:0/1460467787 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4e9c013070 con 0x7f4ea8103090 2024-04-26T21:25:57.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:57 smithi046 ceph-mon[29201]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:25:57.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:25:57.854+0000 7f4eaeb89700 1 -- 172.21.15.46:0/1460467787 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi142", "target": ["mon-mgr", ""]}) -- 0x7f4ea8199bc0 con 0x7f4e980396b0 2024-04-26T21:25:59.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:59 smithi046 ceph-mon[29201]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:25:59.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:59 smithi046 ceph-mon[29201]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi142", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:25:59.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:59 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:25:59.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:59 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:25:59.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:59 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:25:59.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:59 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:25:59.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:59 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi050", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:25:59.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:59 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:59.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:59 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:25:59.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:59 smithi046 ceph-mon[29201]: Updating smithi050:/etc/ceph/ceph.conf 2024-04-26T21:25:59.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:59 smithi046 ceph-mon[29201]: Deploying cephadm binary to smithi142 2024-04-26T21:25:59.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:25:59 smithi046 ceph-mon[29201]: Updating smithi050:/var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:26:00.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:00 smithi046 ceph-mon[29201]: Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:26:01.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:01 smithi046 ceph-mon[29201]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:02.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:02.160+0000 7f4ea5ffb700 1 -- 172.21.15.46:0/1460467787 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f4ea8199bc0 con 0x7f4e980396b0 2024-04-26T21:26:02.162 INFO:teuthology.orchestra.run.smithi046.stdout:Added host 'smithi142' with addr '172.21.15.142' 2024-04-26T21:26:02.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:02.162+0000 7f4eaeb89700 1 -- 172.21.15.46:0/1460467787 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4e980396b0 msgr2=0x7f4e9803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:02.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:02.162+0000 7f4eaeb89700 1 --2- 172.21.15.46:0/1460467787 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4e980396b0 0x7f4e9803bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4e94006fd0 tx=0x7f4e94006e40 comp rx=0 tx=0).stop 2024-04-26T21:26:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:02.162+0000 7f4eaeb89700 1 -- 172.21.15.46:0/1460467787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ea8103090 msgr2=0x7f4ea819e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:02.162+0000 7f4eaeb89700 1 --2- 172.21.15.46:0/1460467787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ea8103090 0x7f4ea819e5c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c0056d0 tx=0x7f4e9c0057b0 comp rx=0 tx=0).stop 2024-04-26T21:26:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:02.162+0000 7f4eaeb89700 1 -- 172.21.15.46:0/1460467787 shutdown_connections 2024-04-26T21:26:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:02.162+0000 7f4eaeb89700 1 --2- 172.21.15.46:0/1460467787 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4e980396b0 0x7f4e9803bb70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:02.162+0000 7f4eaeb89700 1 --2- 172.21.15.46:0/1460467787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ea8103090 0x7f4ea819e5c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:02.162+0000 7f4eaeb89700 1 -- 172.21.15.46:0/1460467787 >> 172.21.15.46:0/1460467787 conn(0x7f4ea80fe940 msgr2=0x7f4ea8104a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:02.162+0000 7f4eaeb89700 1 -- 172.21.15.46:0/1460467787 shutdown_connections 2024-04-26T21:26:02.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:02.163+0000 7f4eaeb89700 1 -- 172.21.15.46:0/1460467787 wait complete. 2024-04-26T21:26:02.776 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-26T21:26:03.018 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:26:03.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:02 smithi046 ceph-mon[29201]: Updating smithi050:/var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T21:26:03.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:02 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:03.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:02 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:03.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:02 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:03.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:02 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:03.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:02 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:03.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:02 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:04.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:03 smithi046 ceph-mon[29201]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:04.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:03 smithi046 ceph-mon[29201]: Added host smithi142 2024-04-26T21:26:04.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:03 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:04.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.565+0000 7f746fc5e700 1 -- 172.21.15.46:0/3565331412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7468103090 msgr2=0x7f7468103470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:04.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.565+0000 7f746fc5e700 1 --2- 172.21.15.46:0/3565331412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7468103090 0x7f7468103470 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7464009910 tx=0x7f7464009c20 comp rx=0 tx=0).stop 2024-04-26T21:26:04.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.565+0000 7f746fc5e700 1 -- 172.21.15.46:0/3565331412 shutdown_connections 2024-04-26T21:26:04.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.565+0000 7f746fc5e700 1 --2- 172.21.15.46:0/3565331412 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7468103090 0x7f7468103470 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:04.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.565+0000 7f746fc5e700 1 -- 172.21.15.46:0/3565331412 >> 172.21.15.46:0/3565331412 conn(0x7f74680fe960 msgr2=0x7f7468100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:04.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.565+0000 7f746fc5e700 1 -- 172.21.15.46:0/3565331412 shutdown_connections 2024-04-26T21:26:04.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.565+0000 7f746fc5e700 1 -- 172.21.15.46:0/3565331412 wait complete. 2024-04-26T21:26:04.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.566+0000 7f746fc5e700 1 Processor -- start 2024-04-26T21:26:04.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.566+0000 7f746fc5e700 1 -- start start 2024-04-26T21:26:04.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.566+0000 7f746fc5e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7468103090 0x7f746819e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:04.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.566+0000 7f746fc5e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f746810d310 con 0x7f7468103090 2024-04-26T21:26:04.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.566+0000 7f746d9fa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7468103090 0x7f746819e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:04.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.566+0000 7f746d9fa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7468103090 0x7f746819e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53158/0 (socket says 172.21.15.46:53158) 2024-04-26T21:26:04.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.566+0000 7f746d9fa700 1 -- 172.21.15.46:0/4213906065 learned_addr learned my addr 172.21.15.46:0/4213906065 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:26:04.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.566+0000 7f746d9fa700 1 -- 172.21.15.46:0/4213906065 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f746819eb20 con 0x7f7468103090 2024-04-26T21:26:04.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.567+0000 7f746d9fa700 1 --2- 172.21.15.46:0/4213906065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7468103090 0x7f746819e5e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f74640056d0 tx=0x7f74640057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:04.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.567+0000 7f745effd700 1 -- 172.21.15.46:0/4213906065 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7464026070 con 0x7f7468103090 2024-04-26T21:26:04.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.567+0000 7f745effd700 1 -- 172.21.15.46:0/4213906065 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f746400bba0 con 0x7f7468103090 2024-04-26T21:26:04.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.567+0000 7f745effd700 1 -- 172.21.15.46:0/4213906065 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7464003920 con 0x7f7468103090 2024-04-26T21:26:04.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.567+0000 7f746fc5e700 1 -- 172.21.15.46:0/4213906065 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f746819ee10 con 0x7f7468103090 2024-04-26T21:26:04.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.567+0000 7f746fc5e700 1 -- 172.21.15.46:0/4213906065 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7468198880 con 0x7f7468103090 2024-04-26T21:26:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.568+0000 7f745effd700 1 -- 172.21.15.46:0/4213906065 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f7464029070 con 0x7f7468103090 2024-04-26T21:26:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.568+0000 7f746fc5e700 1 -- 172.21.15.46:0/4213906065 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7468104570 con 0x7f7468103090 2024-04-26T21:26:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.568+0000 7f745effd700 1 --2- 172.21.15.46:0/4213906065 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f74540396b0 0x7f745403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.568+0000 7f745effd700 1 -- 172.21.15.46:0/4213906065 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f746401e070 con 0x7f7468103090 2024-04-26T21:26:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.568+0000 7f746d1f9700 1 --2- 172.21.15.46:0/4213906065 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f74540396b0 0x7f745403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.569+0000 7f746d1f9700 1 --2- 172.21.15.46:0/4213906065 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f74540396b0 0x7f745403bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7458006fd0 tx=0x7f7458006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:04.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.573+0000 7f745effd700 1 -- 172.21.15.46:0/4213906065 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7464013070 con 0x7f7468103090 2024-04-26T21:26:04.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.787+0000 7f746fc5e700 1 -- 172.21.15.46:0/4213906065 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7468199c80 con 0x7f74540396b0 2024-04-26T21:26:04.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.788+0000 7f745effd700 1 -- 172.21.15.46:0/4213906065 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7f7468199c80 con 0x7f74540396b0 2024-04-26T21:26:04.788 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:26:04.788 INFO:teuthology.orchestra.run.smithi046.stdout:[{"addr": "172.21.15.46", "hostname": "smithi046", "labels": [], "status": ""}, {"addr": "172.21.15.50", "hostname": "smithi050", "labels": [], "status": ""}, {"addr": "172.21.15.142", "hostname": "smithi142", "labels": [], "status": ""}] 2024-04-26T21:26:04.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.790+0000 7f746fc5e700 1 -- 172.21.15.46:0/4213906065 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f74540396b0 msgr2=0x7f745403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:04.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.790+0000 7f746fc5e700 1 --2- 172.21.15.46:0/4213906065 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f74540396b0 0x7f745403bb70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7458006fd0 tx=0x7f7458006e40 comp rx=0 tx=0).stop 2024-04-26T21:26:04.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.790+0000 7f746fc5e700 1 -- 172.21.15.46:0/4213906065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7468103090 msgr2=0x7f746819e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:04.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.790+0000 7f746fc5e700 1 --2- 172.21.15.46:0/4213906065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7468103090 0x7f746819e5e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f74640056d0 tx=0x7f74640057b0 comp rx=0 tx=0).stop 2024-04-26T21:26:04.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.790+0000 7f746fc5e700 1 -- 172.21.15.46:0/4213906065 shutdown_connections 2024-04-26T21:26:04.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.790+0000 7f746fc5e700 1 --2- 172.21.15.46:0/4213906065 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f74540396b0 0x7f745403bb70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:04.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.790+0000 7f746fc5e700 1 --2- 172.21.15.46:0/4213906065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7468103090 0x7f746819e5e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:04.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.790+0000 7f746fc5e700 1 -- 172.21.15.46:0/4213906065 >> 172.21.15.46:0/4213906065 conn(0x7f74680fe960 msgr2=0x7f7468104a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:04.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.790+0000 7f746fc5e700 1 -- 172.21.15.46:0/4213906065 shutdown_connections 2024-04-26T21:26:04.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:04.790+0000 7f746fc5e700 1 -- 172.21.15.46:0/4213906065 wait complete. 2024-04-26T21:26:05.361 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-26T21:26:05.361 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-26T21:26:05.603 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:26:06.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:05 smithi046 ceph-mon[29201]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:06.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:05 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:06.836 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:06 smithi046 ceph-mon[29201]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:26:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.052+0000 7f37f2e32700 1 -- 172.21.15.46:0/3133229147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37ec101fd0 msgr2=0x7f37ec1023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.052+0000 7f37f2e32700 1 --2- 172.21.15.46:0/3133229147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37ec101fd0 0x7f37ec1023b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f37e0009910 tx=0x7f37e0009c20 comp rx=0 tx=0).stop 2024-04-26T21:26:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.052+0000 7f37f2e32700 1 -- 172.21.15.46:0/3133229147 shutdown_connections 2024-04-26T21:26:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.052+0000 7f37f2e32700 1 --2- 172.21.15.46:0/3133229147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37ec101fd0 0x7f37ec1023b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.052+0000 7f37f2e32700 1 -- 172.21.15.46:0/3133229147 >> 172.21.15.46:0/3133229147 conn(0x7f37ec077880 msgr2=0x7f37ec077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.052+0000 7f37f2e32700 1 -- 172.21.15.46:0/3133229147 shutdown_connections 2024-04-26T21:26:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.052+0000 7f37f2e32700 1 -- 172.21.15.46:0/3133229147 wait complete. 2024-04-26T21:26:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.053+0000 7f37f2e32700 1 Processor -- start 2024-04-26T21:26:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.053+0000 7f37f2e32700 1 -- start start 2024-04-26T21:26:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.053+0000 7f37f2e32700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37ec101fd0 0x7f37ec103a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.053+0000 7f37f2e32700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f37ec10c200 con 0x7f37ec101fd0 2024-04-26T21:26:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.053+0000 7f37f0bce700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37ec101fd0 0x7f37ec103a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.053+0000 7f37f0bce700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37ec101fd0 0x7f37ec103a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51930/0 (socket says 172.21.15.46:51930) 2024-04-26T21:26:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.053+0000 7f37f0bce700 1 -- 172.21.15.46:0/208535254 learned_addr learned my addr 172.21.15.46:0/208535254 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:26:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.053+0000 7f37f0bce700 1 -- 172.21.15.46:0/208535254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37ec105910 con 0x7f37ec101fd0 2024-04-26T21:26:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.053+0000 7f37f0bce700 1 --2- 172.21.15.46:0/208535254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37ec101fd0 0x7f37ec103a20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f37e00056d0 tx=0x7f37e00057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.054+0000 7f37e9ffb700 1 -- 172.21.15.46:0/208535254 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37e0026070 con 0x7f37ec101fd0 2024-04-26T21:26:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.054+0000 7f37e9ffb700 1 -- 172.21.15.46:0/208535254 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37e000bc00 con 0x7f37ec101fd0 2024-04-26T21:26:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.054+0000 7f37f2e32700 1 -- 172.21.15.46:0/208535254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37ec103fc0 con 0x7f37ec101fd0 2024-04-26T21:26:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.054+0000 7f37e9ffb700 1 -- 172.21.15.46:0/208535254 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37e0003920 con 0x7f37ec101fd0 2024-04-26T21:26:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.054+0000 7f37f2e32700 1 -- 172.21.15.46:0/208535254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37ec104480 con 0x7f37ec101fd0 2024-04-26T21:26:07.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.055+0000 7f37e9ffb700 1 -- 172.21.15.46:0/208535254 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f37e0029070 con 0x7f37ec101fd0 2024-04-26T21:26:07.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.055+0000 7f37e9ffb700 1 --2- 172.21.15.46:0/208535254 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f37dc039650 0x7f37dc03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:07.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.055+0000 7f37e9ffb700 1 -- 172.21.15.46:0/208535254 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f37e001e070 con 0x7f37ec101fd0 2024-04-26T21:26:07.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.055+0000 7f37f2e32700 1 -- 172.21.15.46:0/208535254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37ec103460 con 0x7f37ec101fd0 2024-04-26T21:26:07.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.055+0000 7f37ebfff700 1 --2- 172.21.15.46:0/208535254 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f37dc039650 0x7f37dc03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:07.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.055+0000 7f37ebfff700 1 --2- 172.21.15.46:0/208535254 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f37dc039650 0x7f37dc03bb10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f37d8006fd0 tx=0x7f37d8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:07.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.060+0000 7f37e9ffb700 1 -- 172.21.15.46:0/208535254 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f37e0030070 con 0x7f37ec101fd0 2024-04-26T21:26:07.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.274+0000 7f37f2e32700 1 -- 172.21.15.46:0/208535254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f37ec104f50 con 0x7f37ec101fd0 2024-04-26T21:26:07.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.769+0000 7f37e9ffb700 1 -- 172.21.15.46:0/208535254 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f37e0013070 con 0x7f37ec101fd0 2024-04-26T21:26:07.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.771+0000 7f37f2e32700 1 -- 172.21.15.46:0/208535254 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f37dc039650 msgr2=0x7f37dc03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:07.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.772+0000 7f37f2e32700 1 --2- 172.21.15.46:0/208535254 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f37dc039650 0x7f37dc03bb10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f37d8006fd0 tx=0x7f37d8006e40 comp rx=0 tx=0).stop 2024-04-26T21:26:07.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.772+0000 7f37f2e32700 1 -- 172.21.15.46:0/208535254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37ec101fd0 msgr2=0x7f37ec103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:07.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.772+0000 7f37f2e32700 1 --2- 172.21.15.46:0/208535254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37ec101fd0 0x7f37ec103a20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f37e00056d0 tx=0x7f37e00057b0 comp rx=0 tx=0).stop 2024-04-26T21:26:07.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.772+0000 7f37f2e32700 1 -- 172.21.15.46:0/208535254 shutdown_connections 2024-04-26T21:26:07.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.772+0000 7f37f2e32700 1 --2- 172.21.15.46:0/208535254 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f37dc039650 0x7f37dc03bb10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:07.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.772+0000 7f37f2e32700 1 --2- 172.21.15.46:0/208535254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37ec101fd0 0x7f37ec103a20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:07.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.772+0000 7f37f2e32700 1 -- 172.21.15.46:0/208535254 >> 172.21.15.46:0/208535254 conn(0x7f37ec077880 msgr2=0x7f37ec06a840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:07.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.772+0000 7f37f2e32700 1 -- 172.21.15.46:0/208535254 shutdown_connections 2024-04-26T21:26:07.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:07.772+0000 7f37f2e32700 1 -- 172.21.15.46:0/208535254 wait complete. 2024-04-26T21:26:07.773 INFO:teuthology.orchestra.run.smithi046.stderr:adjusted tunables profile to default 2024-04-26T21:26:08.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:07 smithi046 ceph-mon[29201]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:08.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:07 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/208535254' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-26T21:26:08.614 INFO:tasks.cephadm:Adding mon.a on smithi046 2024-04-26T21:26:08.614 INFO:tasks.cephadm:Adding mon.b on smithi050 2024-04-26T21:26:08.614 INFO:tasks.cephadm:Adding mon.c on smithi142 2024-04-26T21:26:08.614 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi046:172.21.15.46=a;smithi050:172.21.15.50=b;smithi142:172.21.15.142=c' 2024-04-26T21:26:08.843 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:26:09.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:08 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/208535254' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-26T21:26:09.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:08 smithi046 ceph-mon[29201]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T21:26:09.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:08 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:09.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:08 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:09.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:08 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:09.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:08 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:09.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:08 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi142", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:26:09.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:08 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:09.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:08 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:09.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:08 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:09.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:08 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:09.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:08 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:10.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:09 smithi046 ceph-mon[29201]: Updating smithi142:/etc/ceph/ceph.conf 2024-04-26T21:26:10.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:09 smithi046 ceph-mon[29201]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:10.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:09 smithi046 ceph-mon[29201]: Updating smithi142:/var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:26:10.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:09 smithi046 ceph-mon[29201]: Updating smithi142:/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:26:10.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:09 smithi046 ceph-mon[29201]: Updating smithi142:/var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T21:26:10.209 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.207+0000 7f7206f9e700 1 -- 172.21.15.142:0/1455569437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7200103090 msgr2=0x7f7200103470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:10.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.207+0000 7f7206f9e700 1 --2- 172.21.15.142:0/1455569437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7200103090 0x7f7200103470 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f71f4009910 tx=0x7f71f4009c20 comp rx=0 tx=0).stop 2024-04-26T21:26:10.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.207+0000 7f7206f9e700 1 -- 172.21.15.142:0/1455569437 shutdown_connections 2024-04-26T21:26:10.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.207+0000 7f7206f9e700 1 --2- 172.21.15.142:0/1455569437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7200103090 0x7f7200103470 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:10.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.207+0000 7f7206f9e700 1 -- 172.21.15.142:0/1455569437 >> 172.21.15.142:0/1455569437 conn(0x7f72000fe960 msgr2=0x7f7200100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:10.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.207+0000 7f7206f9e700 1 -- 172.21.15.142:0/1455569437 shutdown_connections 2024-04-26T21:26:10.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.207+0000 7f7206f9e700 1 -- 172.21.15.142:0/1455569437 wait complete. 2024-04-26T21:26:10.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.208+0000 7f7206f9e700 1 Processor -- start 2024-04-26T21:26:10.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.208+0000 7f7206f9e700 1 -- start start 2024-04-26T21:26:10.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.208+0000 7f7206f9e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7200103090 0x7f7200075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:10.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.208+0000 7f7206f9e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f720010d310 con 0x7f7200103090 2024-04-26T21:26:10.210 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.208+0000 7f7204d3a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7200103090 0x7f7200075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:10.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.208+0000 7f7204d3a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7200103090 0x7f7200075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.142:41538/0 (socket says 172.21.15.142:41538) 2024-04-26T21:26:10.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.208+0000 7f7204d3a700 1 -- 172.21.15.142:0/2431790864 learned_addr learned my addr 172.21.15.142:0/2431790864 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-04-26T21:26:10.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.209+0000 7f7204d3a700 1 -- 172.21.15.142:0/2431790864 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7200077610 con 0x7f7200103090 2024-04-26T21:26:10.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.209+0000 7f7204d3a700 1 --2- 172.21.15.142:0/2431790864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7200103090 0x7f7200075720 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f71f40056d0 tx=0x7f71f40057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:10.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.209+0000 7f71fdffb700 1 -- 172.21.15.142:0/2431790864 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f71f4026070 con 0x7f7200103090 2024-04-26T21:26:10.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.209+0000 7f7206f9e700 1 -- 172.21.15.142:0/2431790864 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7200075cc0 con 0x7f7200103090 2024-04-26T21:26:10.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.209+0000 7f71fdffb700 1 -- 172.21.15.142:0/2431790864 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71f400bc00 con 0x7f7200103090 2024-04-26T21:26:10.211 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.209+0000 7f71fdffb700 1 -- 172.21.15.142:0/2431790864 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f71f4003920 con 0x7f7200103090 2024-04-26T21:26:10.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.209+0000 7f7206f9e700 1 -- 172.21.15.142:0/2431790864 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7200076180 con 0x7f7200103090 2024-04-26T21:26:10.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.210+0000 7f7206f9e700 1 -- 172.21.15.142:0/2431790864 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7200104570 con 0x7f7200103090 2024-04-26T21:26:10.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.210+0000 7f71fdffb700 1 -- 172.21.15.142:0/2431790864 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f71f4029070 con 0x7f7200103090 2024-04-26T21:26:10.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.210+0000 7f71fdffb700 1 --2- 172.21.15.142:0/2431790864 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f71f0039650 0x7f71f003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:10.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.210+0000 7f71fdffb700 1 -- 172.21.15.142:0/2431790864 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f71f401e070 con 0x7f7200103090 2024-04-26T21:26:10.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.211+0000 7f71fffff700 1 --2- 172.21.15.142:0/2431790864 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f71f0039650 0x7f71f003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:10.212 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.211+0000 7f71fffff700 1 --2- 172.21.15.142:0/2431790864 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f71f0039650 0x7f71f003bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f71ec006fd0 tx=0x7f71ec006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:10.216 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.216+0000 7f71fdffb700 1 -- 172.21.15.142:0/2431790864 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f71f4013070 con 0x7f7200103090 2024-04-26T21:26:10.427 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.426+0000 7f7206f9e700 1 -- 172.21.15.142:0/2431790864 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi046:172.21.15.46=a;smithi050:172.21.15.50=b;smithi142:172.21.15.142=c", "target": ["mon-mgr", ""]}) -- 0x7f7200076410 con 0x7f71f0039650 2024-04-26T21:26:10.434 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.433+0000 7f71fdffb700 1 -- 172.21.15.142:0/2431790864 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f7200076410 con 0x7f71f0039650 2024-04-26T21:26:10.434 INFO:teuthology.orchestra.run.smithi142.stdout:Scheduled mon update... 2024-04-26T21:26:10.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.435+0000 7f7206f9e700 1 -- 172.21.15.142:0/2431790864 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f71f0039650 msgr2=0x7f71f003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:10.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.435+0000 7f7206f9e700 1 --2- 172.21.15.142:0/2431790864 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f71f0039650 0x7f71f003bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f71ec006fd0 tx=0x7f71ec006e40 comp rx=0 tx=0).stop 2024-04-26T21:26:10.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.435+0000 7f7206f9e700 1 -- 172.21.15.142:0/2431790864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7200103090 msgr2=0x7f7200075720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:10.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.435+0000 7f7206f9e700 1 --2- 172.21.15.142:0/2431790864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7200103090 0x7f7200075720 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f71f40056d0 tx=0x7f71f40057b0 comp rx=0 tx=0).stop 2024-04-26T21:26:10.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.435+0000 7f7206f9e700 1 -- 172.21.15.142:0/2431790864 shutdown_connections 2024-04-26T21:26:10.436 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.435+0000 7f7206f9e700 1 --2- 172.21.15.142:0/2431790864 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f71f0039650 0x7f71f003bb10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:10.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.435+0000 7f7206f9e700 1 --2- 172.21.15.142:0/2431790864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7200103090 0x7f7200075720 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:10.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.435+0000 7f7206f9e700 1 -- 172.21.15.142:0/2431790864 >> 172.21.15.142:0/2431790864 conn(0x7f72000fe960 msgr2=0x7f7200104a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:10.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.435+0000 7f7206f9e700 1 -- 172.21.15.142:0/2431790864 shutdown_connections 2024-04-26T21:26:10.437 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:10.435+0000 7f7206f9e700 1 -- 172.21.15.142:0/2431790864 wait complete. 2024-04-26T21:26:11.009 DEBUG:teuthology.orchestra.run.smithi050:mon.b> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mon.b.service 2024-04-26T21:26:11.012 DEBUG:teuthology.orchestra.run.smithi142:mon.c> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mon.c.service 2024-04-26T21:26:11.014 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T21:26:11.014 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T21:26:11.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:11 smithi046 ceph-mon[29201]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:11.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:11 smithi046 ceph-mon[29201]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi046:172.21.15.46=a;smithi050:172.21.15.50=b;smithi142:172.21.15.142=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:26:11.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:11 smithi046 ceph-mon[29201]: Saving service mon spec with placement smithi046:172.21.15.46=a;smithi050:172.21.15.50=b;smithi142:172.21.15.142=c;count:3 2024-04-26T21:26:11.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:11 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:11.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:11 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:11.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:11 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:11.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:11 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:11.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:11 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:11.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:11 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T21:26:11.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:11 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:11.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:11 smithi046 ceph-mon[29201]: Deploying daemon mon.c on smithi142 2024-04-26T21:26:12.229 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:26:13.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:13 smithi046 ceph-mon[29201]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:13.899 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.897+0000 7fe8537d8700 1 -- 172.21.15.142:0/1486167016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe84c102880 msgr2=0x7fe84c102c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:13.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.897+0000 7fe8537d8700 1 --2- 172.21.15.142:0/1486167016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe84c102880 0x7fe84c102c60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe83c009910 tx=0x7fe83c009c20 comp rx=0 tx=0).stop 2024-04-26T21:26:13.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.897+0000 7fe8537d8700 1 -- 172.21.15.142:0/1486167016 shutdown_connections 2024-04-26T21:26:13.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.897+0000 7fe8537d8700 1 --2- 172.21.15.142:0/1486167016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe84c102880 0x7fe84c102c60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:13.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.897+0000 7fe8537d8700 1 -- 172.21.15.142:0/1486167016 >> 172.21.15.142:0/1486167016 conn(0x7fe84c079350 msgr2=0x7fe84c079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:13.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.897+0000 7fe8537d8700 1 -- 172.21.15.142:0/1486167016 shutdown_connections 2024-04-26T21:26:13.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.897+0000 7fe8537d8700 1 -- 172.21.15.142:0/1486167016 wait complete. 2024-04-26T21:26:13.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.897+0000 7fe8537d8700 1 Processor -- start 2024-04-26T21:26:13.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.897+0000 7fe8537d8700 1 -- start start 2024-04-26T21:26:13.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.898+0000 7fe8537d8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe84c102880 0x7fe84c19e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:13.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.898+0000 7fe8537d8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe84c10c290 con 0x7fe84c102880 2024-04-26T21:26:13.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.898+0000 7fe851574700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe84c102880 0x7fe84c19e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:13.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.898+0000 7fe851574700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe84c102880 0x7fe84c19e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.142:41680/0 (socket says 172.21.15.142:41680) 2024-04-26T21:26:13.900 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.898+0000 7fe851574700 1 -- 172.21.15.142:0/343303617 learned_addr learned my addr 172.21.15.142:0/343303617 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-04-26T21:26:13.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.898+0000 7fe851574700 1 -- 172.21.15.142:0/343303617 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe84c19eb20 con 0x7fe84c102880 2024-04-26T21:26:13.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.898+0000 7fe851574700 1 --2- 172.21.15.142:0/343303617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe84c102880 0x7fe84c19e5e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe83c006010 tx=0x7fe83c005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:13.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.898+0000 7fe8427fc700 1 -- 172.21.15.142:0/343303617 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe83c026070 con 0x7fe84c102880 2024-04-26T21:26:13.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.899+0000 7fe8537d8700 1 -- 172.21.15.142:0/343303617 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe84c19ee10 con 0x7fe84c102880 2024-04-26T21:26:13.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.899+0000 7fe8427fc700 1 -- 172.21.15.142:0/343303617 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe83c00bba0 con 0x7fe84c102880 2024-04-26T21:26:13.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.899+0000 7fe8427fc700 1 -- 172.21.15.142:0/343303617 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe83c003910 con 0x7fe84c102880 2024-04-26T21:26:13.901 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.899+0000 7fe8537d8700 1 -- 172.21.15.142:0/343303617 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe84c198880 con 0x7fe84c102880 2024-04-26T21:26:13.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.899+0000 7fe8537d8700 1 -- 172.21.15.142:0/343303617 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe84c103630 con 0x7fe84c102880 2024-04-26T21:26:13.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.899+0000 7fe8427fc700 1 -- 172.21.15.142:0/343303617 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fe83c029070 con 0x7fe84c102880 2024-04-26T21:26:13.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.900+0000 7fe8427fc700 1 --2- 172.21.15.142:0/343303617 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe838039650 0x7fe83803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:13.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.900+0000 7fe8427fc700 1 -- 172.21.15.142:0/343303617 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe83c01e070 con 0x7fe84c102880 2024-04-26T21:26:13.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.900+0000 7fe850d73700 1 --2- 172.21.15.142:0/343303617 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe838039650 0x7fe83803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:13.902 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.900+0000 7fe850d73700 1 --2- 172.21.15.142:0/343303617 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe838039650 0x7fe83803bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe848006fd0 tx=0x7fe848006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:13.906 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:13.905+0000 7fe8427fc700 1 -- 172.21.15.142:0/343303617 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe83c013070 con 0x7fe84c102880 2024-04-26T21:26:14.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:14.193+0000 7fe8537d8700 1 -- 172.21.15.142:0/343303617 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fe84c199c10 con 0x7fe84c102880 2024-04-26T21:26:14.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:14.193+0000 7fe8427fc700 1 -- 172.21.15.142:0/343303617 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7fe83c04d000 con 0x7fe84c102880 2024-04-26T21:26:14.194 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-04-26T21:26:14.195 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":1,"fsid":"36c7013a-0413-11ef-bc93-c7b262605968","modified":"2024-04-26T21:24:27.425916Z","created":"2024-04-26T21:24:27.425916Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-26T21:26:14.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:14.196+0000 7fe8537d8700 1 -- 172.21.15.142:0/343303617 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe838039650 msgr2=0x7fe83803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:14.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:14.196+0000 7fe8537d8700 1 --2- 172.21.15.142:0/343303617 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe838039650 0x7fe83803bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe848006fd0 tx=0x7fe848006e40 comp rx=0 tx=0).stop 2024-04-26T21:26:14.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:14.196+0000 7fe8537d8700 1 -- 172.21.15.142:0/343303617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe84c102880 msgr2=0x7fe84c19e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:14.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:14.196+0000 7fe8537d8700 1 --2- 172.21.15.142:0/343303617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe84c102880 0x7fe84c19e5e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe83c006010 tx=0x7fe83c005780 comp rx=0 tx=0).stop 2024-04-26T21:26:14.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:14.196+0000 7fe8537d8700 1 -- 172.21.15.142:0/343303617 shutdown_connections 2024-04-26T21:26:14.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:14.196+0000 7fe8537d8700 1 --2- 172.21.15.142:0/343303617 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe838039650 0x7fe83803bb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:14.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:14.196+0000 7fe8537d8700 1 --2- 172.21.15.142:0/343303617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe84c102880 0x7fe84c19e5e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:14.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:14.196+0000 7fe8537d8700 1 -- 172.21.15.142:0/343303617 >> 172.21.15.142:0/343303617 conn(0x7fe84c079350 msgr2=0x7fe84c100bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:14.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:14.196+0000 7fe8537d8700 1 -- 172.21.15.142:0/343303617 shutdown_connections 2024-04-26T21:26:14.197 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:14.196+0000 7fe8537d8700 1 -- 172.21.15.142:0/343303617 wait complete. 2024-04-26T21:26:14.198 INFO:teuthology.orchestra.run.smithi142.stderr:dumped monmap epoch 1 2024-04-26T21:26:14.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:14 smithi046 ceph-mon[29201]: from='client.? 172.21.15.142:0/343303617' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T21:26:15.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:15 smithi046 ceph-mon[29201]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:15.834 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T21:26:15.834 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T21:26:16.155 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:26:17.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:17 smithi046 ceph-mon[29201]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:18.039 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:17 smithi142 systemd[1]: Started Ceph mon.c for 36c7013a-0413-11ef-bc93-c7b262605968. 2024-04-26T21:26:23.210 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:26:23.210 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:26:23.210 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: mon.a calling monitor election 2024-04-26T21:26:23.210 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:23.210 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:26:23.210 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:26:23.210 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: mon.c calling monitor election 2024-04-26T21:26:23.210 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:23.210 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:26:23.210 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:26:23.210 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:23.210 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:26:23.211 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T21:26:23.211 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: monmap epoch 2 2024-04-26T21:26:23.211 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: fsid 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:26:23.211 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: last_changed 2024-04-26T21:26:17.836708+0000 2024-04-26T21:26:23.211 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: created 2024-04-26T21:24:27.425916+0000 2024-04-26T21:26:23.211 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: min_mon_release 19 (squid) 2024-04-26T21:26:23.211 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: election_strategy: 1 2024-04-26T21:26:23.211 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: 0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-04-26T21:26:23.211 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: 1: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.c 2024-04-26T21:26:23.211 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: fsmap 2024-04-26T21:26:23.211 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T21:26:23.211 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: mgrmap e13: x(active, since 53s) 2024-04-26T21:26:23.211 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: overall HEALTH_OK 2024-04-26T21:26:23.211 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:23.211 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:23.212 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:23.212 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T21:26:23.212 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:22 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:23.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:26:23.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:26:23.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: mon.a calling monitor election 2024-04-26T21:26:23.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:23.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:26:23.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:26:23.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: mon.c calling monitor election 2024-04-26T21:26:23.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:23.292 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:26:23.292 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:26:23.292 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:23.292 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:26:23.292 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T21:26:23.292 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: monmap epoch 2 2024-04-26T21:26:23.292 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: fsid 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:26:23.292 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: last_changed 2024-04-26T21:26:17.836708+0000 2024-04-26T21:26:23.292 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: created 2024-04-26T21:24:27.425916+0000 2024-04-26T21:26:23.292 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: min_mon_release 19 (squid) 2024-04-26T21:26:23.292 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: election_strategy: 1 2024-04-26T21:26:23.292 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: 0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-04-26T21:26:23.292 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: 1: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.c 2024-04-26T21:26:23.292 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: fsmap 2024-04-26T21:26:23.292 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T21:26:23.293 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: mgrmap e13: x(active, since 53s) 2024-04-26T21:26:23.293 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: overall HEALTH_OK 2024-04-26T21:26:23.293 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:23.293 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:23.293 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:23.293 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T21:26:23.293 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:22 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:24.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.141+0000 7efd27ae8700 1 -- 172.21.15.142:0/1278861148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c0062a0 msgr2=0x7efd0c008710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:24.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.141+0000 7efd27ae8700 1 --2- 172.21.15.142:0/1278861148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c0062a0 0x7efd0c008710 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7efd20104000 tx=0x7efd1c00fe90 comp rx=0 tx=0).stop 2024-04-26T21:26:24.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.141+0000 7efd27ae8700 1 -- 172.21.15.142:0/1278861148 shutdown_connections 2024-04-26T21:26:24.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.141+0000 7efd27ae8700 1 --2- 172.21.15.142:0/1278861148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c0062a0 0x7efd0c008710 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:24.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.141+0000 7efd27ae8700 1 -- 172.21.15.142:0/1278861148 >> 172.21.15.142:0/1278861148 conn(0x7efd200fea30 msgr2=0x7efd20100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:24.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.141+0000 7efd27ae8700 1 -- 172.21.15.142:0/1278861148 shutdown_connections 2024-04-26T21:26:24.143 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.141+0000 7efd27ae8700 1 -- 172.21.15.142:0/1278861148 wait complete. 2024-04-26T21:26:24.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.142+0000 7efd27ae8700 1 Processor -- start 2024-04-26T21:26:24.145 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.142+0000 7efd27ae8700 1 -- start start 2024-04-26T21:26:24.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.143+0000 7efd27ae8700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c0062a0 0x7efd2019e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:24.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.143+0000 7efd27ae8700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efd2019eca0 0x7efd201989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:24.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.143+0000 7efd27ae8700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efd2010ae70 con 0x7efd0c0062a0 2024-04-26T21:26:24.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.143+0000 7efd27ae8700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efd2010acf0 con 0x7efd2019eca0 2024-04-26T21:26:24.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.143+0000 7efd25083700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efd2019eca0 0x7efd201989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:24.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.143+0000 7efd25884700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c0062a0 0x7efd2019e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:24.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.143+0000 7efd25083700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efd2019eca0 0x7efd201989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:33006/0 (socket says 172.21.15.142:33006) 2024-04-26T21:26:24.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.143+0000 7efd25884700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c0062a0 0x7efd2019e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.142:53370/0 (socket says 172.21.15.142:53370) 2024-04-26T21:26:24.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.143+0000 7efd25083700 1 -- 172.21.15.142:0/1096148903 learned_addr learned my addr 172.21.15.142:0/1096148903 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-04-26T21:26:24.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.143+0000 7efd25083700 1 -- 172.21.15.142:0/1096148903 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efd2019eca0 msgr2=0x7efd201989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-26T21:26:24.146 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.143+0000 7efd25083700 1 -- 172.21.15.142:0/1096148903 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efd2019eca0 msgr2=0x7efd201989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T21:26:24.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.143+0000 7efd25083700 1 --2- 172.21.15.142:0/1096148903 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efd2019eca0 0x7efd201989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T21:26:24.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.143+0000 7efd25083700 1 --2- 172.21.15.142:0/1096148903 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efd2019eca0 0x7efd201989c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T21:26:24.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.143+0000 7efd25884700 1 -- 172.21.15.142:0/1096148903 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efd2019eca0 msgr2=0x7efd201989c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T21:26:24.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.143+0000 7efd25884700 1 --2- 172.21.15.142:0/1096148903 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efd2019eca0 0x7efd201989c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:24.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.143+0000 7efd25884700 1 -- 172.21.15.142:0/1096148903 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd2019f2e0 con 0x7efd0c0062a0 2024-04-26T21:26:24.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.143+0000 7efd25884700 1 --2- 172.21.15.142:0/1096148903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c0062a0 0x7efd2019e760 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7efd1c010010 tx=0x7efd1c01da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:24.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.144+0000 7efd16ffd700 1 -- 172.21.15.142:0/1096148903 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7efd1c005400 con 0x7efd0c0062a0 2024-04-26T21:26:24.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.144+0000 7efd16ffd700 1 -- 172.21.15.142:0/1096148903 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd1c0055a0 con 0x7efd0c0062a0 2024-04-26T21:26:24.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.144+0000 7efd27ae8700 1 -- 172.21.15.142:0/1096148903 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd20198f00 con 0x7efd0c0062a0 2024-04-26T21:26:24.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.144+0000 7efd16ffd700 1 -- 172.21.15.142:0/1096148903 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7efd1c011750 con 0x7efd0c0062a0 2024-04-26T21:26:24.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.144+0000 7efd27ae8700 1 -- 172.21.15.142:0/1096148903 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd20199490 con 0x7efd0c0062a0 2024-04-26T21:26:24.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.144+0000 7efd16ffd700 1 -- 172.21.15.142:0/1096148903 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7efd1c011a00 con 0x7efd0c0062a0 2024-04-26T21:26:24.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.145+0000 7efd16ffd700 1 --2- 172.21.15.142:0/1096148903 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7efd04039440 0x7efd0403b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:24.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.145+0000 7efd16ffd700 1 -- 172.21.15.142:0/1096148903 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7efd1c0505a0 con 0x7efd0c0062a0 2024-04-26T21:26:24.147 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.145+0000 7efd25083700 1 --2- 172.21.15.142:0/1096148903 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7efd04039440 0x7efd0403b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:24.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.145+0000 7efd27ae8700 1 -- 172.21.15.142:0/1096148903 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd20104680 con 0x7efd0c0062a0 2024-04-26T21:26:24.148 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.145+0000 7efd25083700 1 --2- 172.21.15.142:0/1096148903 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7efd04039440 0x7efd0403b900 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efd10009180 tx=0x7efd10008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:24.151 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.151+0000 7efd16ffd700 1 -- 172.21.15.142:0/1096148903 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7efd1c01b2d0 con 0x7efd0c0062a0 2024-04-26T21:26:24.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:23 smithi142 ceph-mon[32944]: Deploying daemon mon.b on smithi050 2024-04-26T21:26:24.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:23 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:26:24.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:23 smithi046 ceph-mon[29201]: Deploying daemon mon.b on smithi050 2024-04-26T21:26:24.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:23 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:26:24.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.438+0000 7efd27ae8700 1 -- 172.21.15.142:0/1096148903 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7efd20199e50 con 0x7efd0c0062a0 2024-04-26T21:26:24.440 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.439+0000 7efd16ffd700 1 -- 172.21.15.142:0/1096148903 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7efd1c010c00 con 0x7efd0c0062a0 2024-04-26T21:26:24.440 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-04-26T21:26:24.440 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":2,"fsid":"36c7013a-0413-11ef-bc93-c7b262605968","modified":"2024-04-26T21:26:17.836708Z","created":"2024-04-26T21:24:27.425916Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-26T21:26:24.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.441+0000 7efd27ae8700 1 -- 172.21.15.142:0/1096148903 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7efd04039440 msgr2=0x7efd0403b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:24.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.441+0000 7efd27ae8700 1 --2- 172.21.15.142:0/1096148903 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7efd04039440 0x7efd0403b900 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efd10009180 tx=0x7efd10008040 comp rx=0 tx=0).stop 2024-04-26T21:26:24.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.441+0000 7efd27ae8700 1 -- 172.21.15.142:0/1096148903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c0062a0 msgr2=0x7efd2019e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:24.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.441+0000 7efd27ae8700 1 --2- 172.21.15.142:0/1096148903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c0062a0 0x7efd2019e760 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7efd1c010010 tx=0x7efd1c01da60 comp rx=0 tx=0).stop 2024-04-26T21:26:24.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.441+0000 7efd27ae8700 1 -- 172.21.15.142:0/1096148903 shutdown_connections 2024-04-26T21:26:24.442 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.441+0000 7efd27ae8700 1 --2- 172.21.15.142:0/1096148903 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7efd04039440 0x7efd0403b900 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:24.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.441+0000 7efd27ae8700 1 --2- 172.21.15.142:0/1096148903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd0c0062a0 0x7efd2019e760 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:24.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.441+0000 7efd27ae8700 1 --2- 172.21.15.142:0/1096148903 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efd2019eca0 0x7efd201989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:24.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.441+0000 7efd27ae8700 1 -- 172.21.15.142:0/1096148903 >> 172.21.15.142:0/1096148903 conn(0x7efd200fea30 msgr2=0x7efd20106b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:24.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.441+0000 7efd27ae8700 1 -- 172.21.15.142:0/1096148903 shutdown_connections 2024-04-26T21:26:24.443 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:24.441+0000 7efd27ae8700 1 -- 172.21.15.142:0/1096148903 wait complete. 2024-04-26T21:26:24.443 INFO:teuthology.orchestra.run.smithi142.stderr:dumped monmap epoch 2 2024-04-26T21:26:25.037 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:24 smithi142 ceph-mon[32944]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:25.037 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:24 smithi142 ceph-mon[32944]: from='client.? 172.21.15.142:0/1096148903' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T21:26:25.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:24 smithi046 ceph-mon[29201]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:25.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:24 smithi046 ceph-mon[29201]: from='client.? 172.21.15.142:0/1096148903' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T21:26:25.304 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:26:24 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:26:24.836+0000 7f94ae7e4700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-26T21:26:26.065 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T21:26:26.066 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T21:26:26.310 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:26:26.862 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: rocksdb: DB pointer 0x55783d0d0000 2024-04-26T21:26:26.862 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-26T21:26:27.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-26T21:26:27.119 INFO:journalctl@ceph.mon.b.smithi050.stdout: ** DB Stats ** 2024-04-26T21:26:27.119 INFO:journalctl@ceph.mon.b.smithi050.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T21:26:27.119 INFO:journalctl@ceph.mon.b.smithi050.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-26T21:26:27.119 INFO:journalctl@ceph.mon.b.smithi050.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T21:26:27.119 INFO:journalctl@ceph.mon.b.smithi050.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T21:26:27.119 INFO:journalctl@ceph.mon.b.smithi050.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-26T21:26:27.119 INFO:journalctl@ceph.mon.b.smithi050.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T21:26:27.119 INFO:journalctl@ceph.mon.b.smithi050.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T21:26:27.119 INFO:journalctl@ceph.mon.b.smithi050.stdout: 2024-04-26T21:26:27.119 INFO:journalctl@ceph.mon.b.smithi050.stdout: ** Compaction Stats [default] ** 2024-04-26T21:26:27.119 INFO:journalctl@ceph.mon.b.smithi050.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T21:26:27.119 INFO:journalctl@ceph.mon.b.smithi050.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-26T21:26:27.120 INFO:journalctl@ceph.mon.b.smithi050.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T21:26:27.120 INFO:journalctl@ceph.mon.b.smithi050.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T21:26:27.120 INFO:journalctl@ceph.mon.b.smithi050.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T21:26:27.120 INFO:journalctl@ceph.mon.b.smithi050.stdout: 2024-04-26T21:26:27.120 INFO:journalctl@ceph.mon.b.smithi050.stdout: ** Compaction Stats [default] ** 2024-04-26T21:26:27.120 INFO:journalctl@ceph.mon.b.smithi050.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T21:26:27.120 INFO:journalctl@ceph.mon.b.smithi050.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-26T21:26:27.120 INFO:journalctl@ceph.mon.b.smithi050.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-26T21:26:27.120 INFO:journalctl@ceph.mon.b.smithi050.stdout: 2024-04-26T21:26:27.120 INFO:journalctl@ceph.mon.b.smithi050.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-26T21:26:27.120 INFO:journalctl@ceph.mon.b.smithi050.stdout: 2024-04-26T21:26:27.120 INFO:journalctl@ceph.mon.b.smithi050.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T21:26:27.120 INFO:journalctl@ceph.mon.b.smithi050.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-26T21:26:27.121 INFO:journalctl@ceph.mon.b.smithi050.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-26T21:26:27.121 INFO:journalctl@ceph.mon.b.smithi050.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-26T21:26:27.121 INFO:journalctl@ceph.mon.b.smithi050.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-26T21:26:27.121 INFO:journalctl@ceph.mon.b.smithi050.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-26T21:26:27.121 INFO:journalctl@ceph.mon.b.smithi050.stdout: Cumulative compaction: 0.00 GB write, 0.13 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T21:26:27.121 INFO:journalctl@ceph.mon.b.smithi050.stdout: Interval compaction: 0.00 GB write, 0.13 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T21:26:27.121 INFO:journalctl@ceph.mon.b.smithi050.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-26T21:26:27.121 INFO:journalctl@ceph.mon.b.smithi050.stdout: Block cache BinnedLRUCache@0x55783cfc71f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-26T21:26:27.121 INFO:journalctl@ceph.mon.b.smithi050.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-26T21:26:27.121 INFO:journalctl@ceph.mon.b.smithi050.stdout: 2024-04-26T21:26:27.121 INFO:journalctl@ceph.mon.b.smithi050.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-26T21:26:27.121 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-26T21:26:27.121 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: using public_addr v2:172.21.15.50:0/0 -> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] 2024-04-26T21:26:27.121 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: starting mon.b rank -1 at public addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] at bind addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:26:27.122 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: mon.b@-1(???) e0 preinit fsid 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:26:27.122 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: mon.b@-1(synchronizing).mds e1 new map 2024-04-26T21:26:27.122 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-26T21:26:27.122 INFO:journalctl@ceph.mon.b.smithi050.stdout: e1 2024-04-26T21:26:27.122 INFO:journalctl@ceph.mon.b.smithi050.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T21:26:27.122 INFO:journalctl@ceph.mon.b.smithi050.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T21:26:27.122 INFO:journalctl@ceph.mon.b.smithi050.stdout: legacy client fscid: -1 2024-04-26T21:26:27.122 INFO:journalctl@ceph.mon.b.smithi050.stdout: 2024-04-26T21:26:27.122 INFO:journalctl@ceph.mon.b.smithi050.stdout: No filesystems configured 2024-04-26T21:26:27.122 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-26T21:26:27.122 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-26T21:26:27.122 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-26T21:26:27.122 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-26T21:26:27.122 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-26T21:26:27.123 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-26T21:26:27.123 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T21:26:27.123 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T21:26:27.123 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T21:26:27.123 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T21:26:27.123 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:27.123 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:27.123 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:27.123 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:26:27.123 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:27.123 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:27.123 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi142", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:26:27.123 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:27.124 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:27.124 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:27.124 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:27.124 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi050", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:26:27.124 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:27.124 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:27.124 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: Updating smithi050:/etc/ceph/ceph.conf 2024-04-26T21:26:27.124 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: Deploying cephadm binary to smithi142 2024-04-26T21:26:27.124 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: Updating smithi050:/var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:26:27.124 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:26:27.124 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:27.124 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: Updating smithi050:/var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T21:26:27.124 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:27.124 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:27.125 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:27.125 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:27.125 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:27.125 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:27.125 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:27.125 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: Added host smithi142 2024-04-26T21:26:27.125 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:27.125 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:27.125 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:27.125 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:26:27.125 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:27.125 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/208535254' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-26T21:26:27.125 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/208535254' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-26T21:26:27.125 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T21:26:27.125 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:27.126 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:27.126 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:27.126 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:27.126 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi142", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:26:27.126 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:27.126 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:27.127 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:27.127 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:27.127 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:27.127 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: Updating smithi142:/etc/ceph/ceph.conf 2024-04-26T21:26:27.127 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:27.127 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: Updating smithi142:/var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:26:27.127 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: Updating smithi142:/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:26:27.127 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: Updating smithi142:/var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T21:26:27.127 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:27.127 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi046:172.21.15.46=a;smithi050:172.21.15.50=b;smithi142:172.21.15.142=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:26:27.127 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: Saving service mon spec with placement smithi046:172.21.15.46=a;smithi050:172.21.15.50=b;smithi142:172.21.15.142=c;count:3 2024-04-26T21:26:27.127 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:27.127 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:27.128 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:27.128 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:27.128 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:27.128 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T21:26:27.128 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:27.128 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: Deploying daemon mon.c on smithi142 2024-04-26T21:26:27.128 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:27.128 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='client.? 172.21.15.142:0/343303617' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T21:26:27.128 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:27.128 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:27.128 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:26:27.128 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:26:27.128 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: mon.a calling monitor election 2024-04-26T21:26:27.128 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:27.128 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:26:27.129 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:26:27.129 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: mon.c calling monitor election 2024-04-26T21:26:27.129 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:27.129 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:26:27.129 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:26:27.129 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:27.129 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:26:27.129 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T21:26:27.129 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: monmap epoch 2 2024-04-26T21:26:27.129 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: fsid 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:26:27.129 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: last_changed 2024-04-26T21:26:17.836708+0000 2024-04-26T21:26:27.129 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: created 2024-04-26T21:24:27.425916+0000 2024-04-26T21:26:27.129 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: min_mon_release 19 (squid) 2024-04-26T21:26:27.129 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: election_strategy: 1 2024-04-26T21:26:27.130 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: 0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-04-26T21:26:27.130 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: 1: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.c 2024-04-26T21:26:27.130 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: fsmap 2024-04-26T21:26:27.130 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T21:26:27.130 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: mgrmap e13: x(active, since 53s) 2024-04-26T21:26:27.130 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: overall HEALTH_OK 2024-04-26T21:26:27.130 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:27.130 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:27.130 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:27.130 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T21:26:27.130 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:27.130 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: Deploying daemon mon.b on smithi050 2024-04-26T21:26:27.130 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:26:27.130 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:27.131 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: from='client.? 172.21.15.142:0/1096148903' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T21:26:27.131 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 ceph-mon[32753]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-26T21:26:27.131 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 bash[32734]: 7e5f775ded483569d34b8a00c3092be900247cc9ceeaf5ca339fcde440741090 2024-04-26T21:26:27.131 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:26 smithi050 systemd[1]: Started Ceph mon.b for 36c7013a-0413-11ef-bc93-c7b262605968. 2024-04-26T21:26:32.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.188+0000 7f600ed59700 1 -- 172.21.15.142:0/2501549816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ff80035c0 msgr2=0x7f5ff8005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:32.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.188+0000 7f600ed59700 1 --2- 172.21.15.142:0/2501549816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ff80035c0 0x7f5ff8005a50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6008102e70 tx=0x7f5ffc00fe60 comp rx=0 tx=0).stop 2024-04-26T21:26:32.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.188+0000 7f600ed59700 1 -- 172.21.15.142:0/2501549816 shutdown_connections 2024-04-26T21:26:32.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.188+0000 7f600ed59700 1 --2- 172.21.15.142:0/2501549816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5ff80035c0 0x7f5ff8005a50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:32.191 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.188+0000 7f600ed59700 1 -- 172.21.15.142:0/2501549816 >> 172.21.15.142:0/2501549816 conn(0x7f60080fea30 msgr2=0x7f6008100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:32.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.188+0000 7f600ed59700 1 -- 172.21.15.142:0/2501549816 shutdown_connections 2024-04-26T21:26:32.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.188+0000 7f600ed59700 1 -- 172.21.15.142:0/2501549816 wait complete. 2024-04-26T21:26:32.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.189+0000 7f600ed59700 1 Processor -- start 2024-04-26T21:26:32.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.189+0000 7f600ed59700 1 -- start start 2024-04-26T21:26:32.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.189+0000 7f600ed59700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ff80035c0 0x7f6008075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:32.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.189+0000 7f600ed59700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6008075c60 0x7f60080760e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:32.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.189+0000 7f600ed59700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60081a4ce0 0x7f60081a70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:32.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.189+0000 7f600ed59700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f600810ae30 con 0x7f60081a4ce0 2024-04-26T21:26:32.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.189+0000 7f600ed59700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f600810acb0 con 0x7f6008075c60 2024-04-26T21:26:32.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.189+0000 7f600ed59700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f600810afb0 con 0x7f5ff80035c0 2024-04-26T21:26:32.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.189+0000 7f600caf5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ff80035c0 0x7f6008075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:32.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.189+0000 7f600caf5700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ff80035c0 0x7f6008075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:33020/0 (socket says 172.21.15.142:33020) 2024-04-26T21:26:32.192 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.189+0000 7f600caf5700 1 -- 172.21.15.142:0/2945330213 learned_addr learned my addr 172.21.15.142:0/2945330213 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-04-26T21:26:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.189+0000 7f6007fff700 1 --2- 172.21.15.142:0/2945330213 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6008075c60 0x7f60080760e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.189+0000 7f600d2f6700 1 --2- 172.21.15.142:0/2945330213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60081a4ce0 0x7f60081a70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.189+0000 7f600caf5700 1 -- 172.21.15.142:0/2945330213 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ff80035c0 msgr2=0x7f6008075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T21:26:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.189+0000 7f600caf5700 1 -- 172.21.15.142:0/2945330213 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ff80035c0 msgr2=0x7f6008075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T21:26:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.189+0000 7f600caf5700 1 --2- 172.21.15.142:0/2945330213 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ff80035c0 0x7f6008075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T21:26:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.189+0000 7f600caf5700 1 --2- 172.21.15.142:0/2945330213 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ff80035c0 0x7f6008075720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T21:26:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.189+0000 7f6007fff700 1 -- 172.21.15.142:0/2945330213 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6008075c60 msgr2=0x7f60080760e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-26T21:26:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.189+0000 7f6007fff700 1 -- 172.21.15.142:0/2945330213 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6008075c60 msgr2=0x7f60080760e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T21:26:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.189+0000 7f6007fff700 1 --2- 172.21.15.142:0/2945330213 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6008075c60 0x7f60080760e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T21:26:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.189+0000 7f6007fff700 1 --2- 172.21.15.142:0/2945330213 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6008075c60 0x7f60080760e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T21:26:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.190+0000 7f600d2f6700 1 -- 172.21.15.142:0/2945330213 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ff80035c0 msgr2=0x7f6008075720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T21:26:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.190+0000 7f600d2f6700 1 --2- 172.21.15.142:0/2945330213 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ff80035c0 0x7f6008075720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.190+0000 7f600d2f6700 1 -- 172.21.15.142:0/2945330213 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6008075c60 msgr2=0x7f60080760e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T21:26:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.190+0000 7f600d2f6700 1 --2- 172.21.15.142:0/2945330213 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6008075c60 0x7f60080760e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:32.193 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.190+0000 7f600d2f6700 1 -- 172.21.15.142:0/2945330213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60081a77a0 con 0x7f60081a4ce0 2024-04-26T21:26:32.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.190+0000 7f600d2f6700 1 --2- 172.21.15.142:0/2945330213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60081a4ce0 0x7f60081a70d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f600000bd50 tx=0x7f600000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:32.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.190+0000 7f6005ffb700 1 -- 172.21.15.142:0/2945330213 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6000019070 con 0x7f60081a4ce0 2024-04-26T21:26:32.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.190+0000 7f6005ffb700 1 -- 172.21.15.142:0/2945330213 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f600000cd10 con 0x7f60081a4ce0 2024-04-26T21:26:32.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.190+0000 7f6005ffb700 1 -- 172.21.15.142:0/2945330213 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6000012700 con 0x7f60081a4ce0 2024-04-26T21:26:32.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.190+0000 7f600ed59700 1 -- 172.21.15.142:0/2945330213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60081a7a90 con 0x7f60081a4ce0 2024-04-26T21:26:32.194 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.190+0000 7f600ed59700 1 -- 172.21.15.142:0/2945330213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60081a7f80 con 0x7f60081a4ce0 2024-04-26T21:26:32.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.191+0000 7f6005ffb700 1 -- 172.21.15.142:0/2945330213 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f600001c070 con 0x7f60081a4ce0 2024-04-26T21:26:32.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.191+0000 7f6005ffb700 1 --2- 172.21.15.142:0/2945330213 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5fec039ae0 0x7f5fec03bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:32.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.191+0000 7f6005ffb700 1 -- 172.21.15.142:0/2945330213 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f600004c0d0 con 0x7f60081a4ce0 2024-04-26T21:26:32.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.191+0000 7f600caf5700 1 --2- 172.21.15.142:0/2945330213 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5fec039ae0 0x7f5fec03bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:32.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.191+0000 7f600ed59700 1 -- 172.21.15.142:0/2945330213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6008064a80 con 0x7f60081a4ce0 2024-04-26T21:26:32.195 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.192+0000 7f600caf5700 1 --2- 172.21.15.142:0/2945330213 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5fec039ae0 0x7f5fec03bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc00fa10 tx=0x7f5ffc005c60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:32.198 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.197+0000 7f6005ffb700 1 -- 172.21.15.142:0/2945330213 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f600000e0a0 con 0x7f60081a4ce0 2024-04-26T21:26:32.198 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:26:32.199 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: mon.a calling monitor election 2024-04-26T21:26:32.199 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: mon.c calling monitor election 2024-04-26T21:26:32.199 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:26:32.199 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:26:32.199 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:32.199 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:26:32.199 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:26:32.199 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: mon.b calling monitor election 2024-04-26T21:26:32.199 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:32.199 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:26:32.199 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:26:32.199 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:26:32.199 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T21:26:32.199 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: monmap epoch 3 2024-04-26T21:26:32.200 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: fsid 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:26:32.200 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: last_changed 2024-04-26T21:26:26.883530+0000 2024-04-26T21:26:32.200 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: created 2024-04-26T21:24:27.425916+0000 2024-04-26T21:26:32.200 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: min_mon_release 19 (squid) 2024-04-26T21:26:32.200 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: election_strategy: 1 2024-04-26T21:26:32.200 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: 0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-04-26T21:26:32.200 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: 1: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.c 2024-04-26T21:26:32.200 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: 2: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2024-04-26T21:26:32.200 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: fsmap 2024-04-26T21:26:32.200 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T21:26:32.200 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: mgrmap e13: x(active, since 62s) 2024-04-26T21:26:32.200 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: overall HEALTH_OK 2024-04-26T21:26:32.200 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:32.200 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:32.200 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:32.201 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:32.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:26:32.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: mon.a calling monitor election 2024-04-26T21:26:32.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: mon.c calling monitor election 2024-04-26T21:26:32.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:26:32.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:26:32.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:32.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:26:32.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:26:32.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: mon.b calling monitor election 2024-04-26T21:26:32.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:32.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:26:32.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:26:32.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:26:32.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T21:26:32.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: monmap epoch 3 2024-04-26T21:26:32.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: fsid 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:26:32.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: last_changed 2024-04-26T21:26:26.883530+0000 2024-04-26T21:26:32.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: created 2024-04-26T21:24:27.425916+0000 2024-04-26T21:26:32.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: min_mon_release 19 (squid) 2024-04-26T21:26:32.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: election_strategy: 1 2024-04-26T21:26:32.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: 0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-04-26T21:26:32.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: 1: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.c 2024-04-26T21:26:32.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: 2: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2024-04-26T21:26:32.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: fsmap 2024-04-26T21:26:32.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T21:26:32.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: mgrmap e13: x(active, since 62s) 2024-04-26T21:26:32.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: overall HEALTH_OK 2024-04-26T21:26:32.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:32.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:32.306 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:32.306 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:32.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:26:32.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: mon.a calling monitor election 2024-04-26T21:26:32.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: mon.c calling monitor election 2024-04-26T21:26:32.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:26:32.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:26:32.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:32.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:26:32.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:26:32.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: mon.b calling monitor election 2024-04-26T21:26:32.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:32.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:26:32.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:26:32.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:26:32.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T21:26:32.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: monmap epoch 3 2024-04-26T21:26:32.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: fsid 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:26:32.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: last_changed 2024-04-26T21:26:26.883530+0000 2024-04-26T21:26:32.370 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: created 2024-04-26T21:24:27.425916+0000 2024-04-26T21:26:32.370 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: min_mon_release 19 (squid) 2024-04-26T21:26:32.370 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: election_strategy: 1 2024-04-26T21:26:32.370 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: 0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-04-26T21:26:32.370 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: 1: [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] mon.c 2024-04-26T21:26:32.370 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: 2: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2024-04-26T21:26:32.370 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: fsmap 2024-04-26T21:26:32.370 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T21:26:32.370 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: mgrmap e13: x(active, since 62s) 2024-04-26T21:26:32.370 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: overall HEALTH_OK 2024-04-26T21:26:32.370 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:32.370 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:32.370 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:32.370 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:32.485 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.484+0000 7f600ed59700 1 -- 172.21.15.142:0/2945330213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f60081a8210 con 0x7f60081a4ce0 2024-04-26T21:26:32.486 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.485+0000 7f6005ffb700 1 -- 172.21.15.142:0/2945330213 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1281 (secure 0 0 0) 0x7f6000007290 con 0x7f60081a4ce0 2024-04-26T21:26:32.486 INFO:teuthology.orchestra.run.smithi142.stdout: 2024-04-26T21:26:32.486 INFO:teuthology.orchestra.run.smithi142.stdout:{"epoch":3,"fsid":"36c7013a-0413-11ef-bc93-c7b262605968","modified":"2024-04-26T21:26:26.883530Z","created":"2024-04-26T21:24:27.425916Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:3300","nonce":0},{"type":"v1","addr":"172.21.15.142:6789","nonce":0}]},"addr":"172.21.15.142:6789/0","public_addr":"172.21.15.142:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-04-26T21:26:32.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.487+0000 7f600ed59700 1 -- 172.21.15.142:0/2945330213 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5fec039ae0 msgr2=0x7f5fec03bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:32.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.487+0000 7f600ed59700 1 --2- 172.21.15.142:0/2945330213 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5fec039ae0 0x7f5fec03bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc00fa10 tx=0x7f5ffc005c60 comp rx=0 tx=0).stop 2024-04-26T21:26:32.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.487+0000 7f600ed59700 1 -- 172.21.15.142:0/2945330213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60081a4ce0 msgr2=0x7f60081a70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:32.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.487+0000 7f600ed59700 1 --2- 172.21.15.142:0/2945330213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60081a4ce0 0x7f60081a70d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f600000bd50 tx=0x7f600000be30 comp rx=0 tx=0).stop 2024-04-26T21:26:32.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.487+0000 7f600ed59700 1 -- 172.21.15.142:0/2945330213 shutdown_connections 2024-04-26T21:26:32.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.487+0000 7f600ed59700 1 --2- 172.21.15.142:0/2945330213 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5fec039ae0 0x7f5fec03bfa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:32.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.487+0000 7f600ed59700 1 --2- 172.21.15.142:0/2945330213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60081a4ce0 0x7f60081a70d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:32.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.487+0000 7f600ed59700 1 --2- 172.21.15.142:0/2945330213 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5ff80035c0 0x7f6008075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:32.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.487+0000 7f600ed59700 1 --2- 172.21.15.142:0/2945330213 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6008075c60 0x7f60080760e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:32.489 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.488+0000 7f600ed59700 1 -- 172.21.15.142:0/2945330213 >> 172.21.15.142:0/2945330213 conn(0x7f60080fea30 msgr2=0x7f6008106b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:32.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.488+0000 7f600ed59700 1 -- 172.21.15.142:0/2945330213 shutdown_connections 2024-04-26T21:26:32.490 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:32.488+0000 7f600ed59700 1 -- 172.21.15.142:0/2945330213 wait complete. 2024-04-26T21:26:32.490 INFO:teuthology.orchestra.run.smithi142.stderr:dumped monmap epoch 3 2024-04-26T21:26:33.216 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:32 smithi142 ceph-mon[32944]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:33.217 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:32 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:33.217 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:32 smithi142 ceph-mon[32944]: from='client.? 172.21.15.142:0/2945330213' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T21:26:33.217 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:32 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:26:33.243 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-26T21:26:33.244 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-26T21:26:33.271 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:32 smithi046 ceph-mon[29201]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:33.272 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:32 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:33.272 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:32 smithi046 ceph-mon[29201]: from='client.? 172.21.15.142:0/2945330213' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T21:26:33.272 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:32 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:26:33.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:32 smithi050 ceph-mon[32753]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:33.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:32 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:33.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:32 smithi050 ceph-mon[32753]: from='client.? 172.21.15.142:0/2945330213' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T21:26:33.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:32 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:26:33.499 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:26:34.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:34 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:34.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:34 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:34.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:34 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:34.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:34 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:34.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:34 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:34.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:34 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:34.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:34 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:34.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:34 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:34.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:34 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:34.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:34 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:34.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:34 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:34.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:34 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:34.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:34 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:34.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:34 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:34.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:34 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:34.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:34 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:34.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:34 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:34.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:34 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:35.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.259+0000 7f4344945700 1 -- 172.21.15.46:0/673440939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4340101fd0 msgr2=0x7f43401023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:35.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.259+0000 7f4344945700 1 --2- 172.21.15.46:0/673440939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4340101fd0 0x7f43401023b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4330009910 tx=0x7f4330009c20 comp rx=0 tx=0).stop 2024-04-26T21:26:35.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.259+0000 7f4344945700 1 -- 172.21.15.46:0/673440939 shutdown_connections 2024-04-26T21:26:35.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.259+0000 7f4344945700 1 --2- 172.21.15.46:0/673440939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4340101fd0 0x7f43401023b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:35.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.259+0000 7f4344945700 1 -- 172.21.15.46:0/673440939 >> 172.21.15.46:0/673440939 conn(0x7f4340077880 msgr2=0x7f4340077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:35.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.260+0000 7f4344945700 1 -- 172.21.15.46:0/673440939 shutdown_connections 2024-04-26T21:26:35.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.260+0000 7f4344945700 1 -- 172.21.15.46:0/673440939 wait complete. 2024-04-26T21:26:35.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.260+0000 7f4344945700 1 Processor -- start 2024-04-26T21:26:35.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.260+0000 7f4344945700 1 -- start start 2024-04-26T21:26:35.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.261+0000 7f4344945700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4340101fd0 0x7f4340103a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:35.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.261+0000 7f4344945700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4340103f60 0x7f43401043e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:35.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.261+0000 7f4344945700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43401a4d90 0x7f43401a7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:35.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.261+0000 7f4344945700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f434010c200 con 0x7f4340103f60 2024-04-26T21:26:35.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.261+0000 7f4344945700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f434010c080 con 0x7f4340101fd0 2024-04-26T21:26:35.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.261+0000 7f4344945700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f434010c380 con 0x7f43401a4d90 2024-04-26T21:26:35.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.261+0000 7f433dd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4340103f60 0x7f43401043e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:35.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.261+0000 7f433dd9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4340103f60 0x7f43401043e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43868/0 (socket says 172.21.15.46:43868) 2024-04-26T21:26:35.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.261+0000 7f433dd9b700 1 -- 172.21.15.46:0/3802866184 learned_addr learned my addr 172.21.15.46:0/3802866184 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:26:35.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.261+0000 7f433e59c700 1 --2- 172.21.15.46:0/3802866184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4340101fd0 0x7f4340103a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:35.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.261+0000 7f433ed9d700 1 --2- 172.21.15.46:0/3802866184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43401a4d90 0x7f43401a7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:35.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.261+0000 7f433dd9b700 1 -- 172.21.15.46:0/3802866184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43401a4d90 msgr2=0x7f43401a7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:35.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.261+0000 7f433dd9b700 1 --2- 172.21.15.46:0/3802866184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43401a4d90 0x7f43401a7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:35.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.261+0000 7f433dd9b700 1 -- 172.21.15.46:0/3802866184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4340101fd0 msgr2=0x7f4340103a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:35.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.261+0000 7f433dd9b700 1 --2- 172.21.15.46:0/3802866184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4340101fd0 0x7f4340103a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:35.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.261+0000 7f433dd9b700 1 -- 172.21.15.46:0/3802866184 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43401a7850 con 0x7f4340103f60 2024-04-26T21:26:35.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.261+0000 7f433dd9b700 1 --2- 172.21.15.46:0/3802866184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4340103f60 0x7f43401043e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f433400b780 tx=0x7f433400bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:35.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.262+0000 7f432f7fe700 1 -- 172.21.15.46:0/3802866184 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4334004230 con 0x7f4340103f60 2024-04-26T21:26:35.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.262+0000 7f432f7fe700 1 -- 172.21.15.46:0/3802866184 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43340043d0 con 0x7f4340103f60 2024-04-26T21:26:35.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.262+0000 7f4344945700 1 -- 172.21.15.46:0/3802866184 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43401a7b40 con 0x7f4340103f60 2024-04-26T21:26:35.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.262+0000 7f432f7fe700 1 -- 172.21.15.46:0/3802866184 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4334011b70 con 0x7f4340103f60 2024-04-26T21:26:35.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.262+0000 7f4344945700 1 -- 172.21.15.46:0/3802866184 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43401a80d0 con 0x7f4340103f60 2024-04-26T21:26:35.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.263+0000 7f432f7fe700 1 -- 172.21.15.46:0/3802866184 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f433401a400 con 0x7f4340103f60 2024-04-26T21:26:35.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.263+0000 7f4344945700 1 -- 172.21.15.46:0/3802866184 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4340064a80 con 0x7f4340103f60 2024-04-26T21:26:35.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.263+0000 7f432f7fe700 1 --2- 172.21.15.46:0/3802866184 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4328039ae0 0x7f432803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:35.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.263+0000 7f432f7fe700 1 -- 172.21.15.46:0/3802866184 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f433404c3d0 con 0x7f4340103f60 2024-04-26T21:26:35.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.263+0000 7f433e59c700 1 --2- 172.21.15.46:0/3802866184 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4328039ae0 0x7f432803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:35.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.264+0000 7f433e59c700 1 --2- 172.21.15.46:0/3802866184 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4328039ae0 0x7f432803bfa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4330009750 tx=0x7f433001a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:35.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.268+0000 7f432f7fe700 1 -- 172.21.15.46:0/3802866184 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4334018750 con 0x7f4340103f60 2024-04-26T21:26:35.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.478+0000 7f4344945700 1 -- 172.21.15.46:0/3802866184 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f43401a83a0 con 0x7f4340103f60 2024-04-26T21:26:35.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.478+0000 7f432f7fe700 1 -- 172.21.15.46:0/3802866184 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7f433401a7f0 con 0x7f4340103f60 2024-04-26T21:26:35.480 INFO:teuthology.orchestra.run.smithi046.stdout:# minimal ceph.conf for 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:26:35.480 INFO:teuthology.orchestra.run.smithi046.stdout:[global] 2024-04-26T21:26:35.480 INFO:teuthology.orchestra.run.smithi046.stdout: fsid = 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:26:35.480 INFO:teuthology.orchestra.run.smithi046.stdout: mon_host = [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] 2024-04-26T21:26:35.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.480+0000 7f4344945700 1 -- 172.21.15.46:0/3802866184 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4328039ae0 msgr2=0x7f432803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:35.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.480+0000 7f4344945700 1 --2- 172.21.15.46:0/3802866184 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4328039ae0 0x7f432803bfa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4330009750 tx=0x7f433001a040 comp rx=0 tx=0).stop 2024-04-26T21:26:35.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.480+0000 7f4344945700 1 -- 172.21.15.46:0/3802866184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4340103f60 msgr2=0x7f43401043e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:35.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.480+0000 7f4344945700 1 --2- 172.21.15.46:0/3802866184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4340103f60 0x7f43401043e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f433400b780 tx=0x7f433400bb40 comp rx=0 tx=0).stop 2024-04-26T21:26:35.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.480+0000 7f4344945700 1 -- 172.21.15.46:0/3802866184 shutdown_connections 2024-04-26T21:26:35.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.480+0000 7f4344945700 1 --2- 172.21.15.46:0/3802866184 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f43401a4d90 0x7f43401a7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:35.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.480+0000 7f4344945700 1 --2- 172.21.15.46:0/3802866184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4340101fd0 0x7f4340103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:35.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.480+0000 7f4344945700 1 --2- 172.21.15.46:0/3802866184 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4328039ae0 0x7f432803bfa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:35.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.480+0000 7f4344945700 1 --2- 172.21.15.46:0/3802866184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4340103f60 0x7f43401043e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:35.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.480+0000 7f4344945700 1 -- 172.21.15.46:0/3802866184 >> 172.21.15.46:0/3802866184 conn(0x7f4340077880 msgr2=0x7f434006a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:35.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.480+0000 7f4344945700 1 -- 172.21.15.46:0/3802866184 shutdown_connections 2024-04-26T21:26:35.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:35.480+0000 7f4344945700 1 -- 172.21.15.46:0/3802866184 wait complete. 2024-04-26T21:26:35.784 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:35 smithi046 ceph-mon[29201]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:35.785 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:35 smithi046 ceph-mon[29201]: Updating smithi046:/etc/ceph/ceph.conf 2024-04-26T21:26:35.785 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:35 smithi046 ceph-mon[29201]: Updating smithi050:/etc/ceph/ceph.conf 2024-04-26T21:26:35.785 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:35 smithi046 ceph-mon[29201]: Updating smithi142:/etc/ceph/ceph.conf 2024-04-26T21:26:35.785 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:35 smithi046 ceph-mon[29201]: Updating smithi050:/var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:26:35.785 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:35 smithi046 ceph-mon[29201]: Updating smithi142:/var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:26:35.785 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:35 smithi046 ceph-mon[29201]: Updating smithi046:/var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:26:35.785 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:35.785 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:35.785 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:35.785 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:35.785 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:35.785 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:35.786 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:35.786 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:35.786 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:35.786 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:35.786 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:35.786 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T21:26:35.786 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T21:26:35.786 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:35.786 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:35.786 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:35 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3802866184' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:36.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:35 smithi142 ceph-mon[32944]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:36.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:35 smithi142 ceph-mon[32944]: Updating smithi046:/etc/ceph/ceph.conf 2024-04-26T21:26:36.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:35 smithi142 ceph-mon[32944]: Updating smithi050:/etc/ceph/ceph.conf 2024-04-26T21:26:36.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:35 smithi142 ceph-mon[32944]: Updating smithi142:/etc/ceph/ceph.conf 2024-04-26T21:26:36.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:35 smithi142 ceph-mon[32944]: Updating smithi050:/var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:26:36.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:35 smithi142 ceph-mon[32944]: Updating smithi142:/var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:26:36.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:35 smithi142 ceph-mon[32944]: Updating smithi046:/var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:26:36.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:36.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:36.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:36.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:36.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:36.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:36.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:36.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:36.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:36.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:36.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:36.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T21:26:36.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T21:26:36.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:36.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:36.043 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:35 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3802866184' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:36.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:35 smithi050 ceph-mon[32753]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:36.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:35 smithi050 ceph-mon[32753]: Updating smithi046:/etc/ceph/ceph.conf 2024-04-26T21:26:36.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:35 smithi050 ceph-mon[32753]: Updating smithi050:/etc/ceph/ceph.conf 2024-04-26T21:26:36.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:35 smithi050 ceph-mon[32753]: Updating smithi142:/etc/ceph/ceph.conf 2024-04-26T21:26:36.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:35 smithi050 ceph-mon[32753]: Updating smithi050:/var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:26:36.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:35 smithi050 ceph-mon[32753]: Updating smithi142:/var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:26:36.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:35 smithi050 ceph-mon[32753]: Updating smithi046:/var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:26:36.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:36.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:36.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:36.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:36.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:36.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:36.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:36.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:36.120 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:36.120 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:36.120 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:36.120 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T21:26:36.120 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T21:26:36.120 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:36.120 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:36.120 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:35 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3802866184' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:36.545 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-26T21:26:36.545 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-26T21:26:36.546 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T21:26:36.614 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-26T21:26:36.614 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:26:36.679 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T21:26:36.679 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T21:26:36.711 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T21:26:36.711 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:26:36.780 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-04-26T21:26:36.781 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T21:26:36.812 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-04-26T21:26:36.812 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:26:36.879 INFO:tasks.cephadm:Adding mgr.x on smithi046 2024-04-26T21:26:36.879 INFO:tasks.cephadm:Adding mgr.y on smithi050 2024-04-26T21:26:36.879 INFO:tasks.cephadm:Adding mgr.z on smithi142 2024-04-26T21:26:36.879 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi046=x;smithi050=y;smithi142=z' 2024-04-26T21:26:36.940 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:36 smithi142 ceph-mon[32944]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T21:26:36.941 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:36 smithi142 ceph-mon[32944]: Reconfiguring daemon mon.a on smithi046 2024-04-26T21:26:36.974 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:36 smithi050 ceph-mon[32753]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T21:26:36.974 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:36 smithi050 ceph-mon[32753]: Reconfiguring daemon mon.a on smithi046 2024-04-26T21:26:37.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:36 smithi046 ceph-mon[29201]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T21:26:37.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:36 smithi046 ceph-mon[29201]: Reconfiguring daemon mon.a on smithi046 2024-04-26T21:26:37.149 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:26:37.899 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:37 smithi050 ceph-mon[32753]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:37.899 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:37 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:37.899 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:37 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:37.899 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:37 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T21:26:37.899 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:37 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T21:26:37.899 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:37 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:38.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:37 smithi142 ceph-mon[32944]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:38.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:37 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:38.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:37 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:38.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:37 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T21:26:38.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:37 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T21:26:38.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:37 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:38.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:37 smithi046 ceph-mon[29201]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:38.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:37 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:38.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:37 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:38.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:37 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T21:26:38.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:37 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T21:26:38.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:37 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:38.687 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.684+0000 7f8f09c29700 1 -- 172.21.15.142:0/1345914970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f04102fb0 msgr2=0x7f8f041033d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:38.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.684+0000 7f8f09c29700 1 --2- 172.21.15.142:0/1345914970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f04102fb0 0x7f8f041033d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4009910 tx=0x7f8ef4009c20 comp rx=0 tx=0).stop 2024-04-26T21:26:38.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.684+0000 7f8f09c29700 1 -- 172.21.15.142:0/1345914970 shutdown_connections 2024-04-26T21:26:38.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.684+0000 7f8f09c29700 1 --2- 172.21.15.142:0/1345914970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f04102fb0 0x7f8f041033d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:38.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.684+0000 7f8f09c29700 1 -- 172.21.15.142:0/1345914970 >> 172.21.15.142:0/1345914970 conn(0x7f8f040fe550 msgr2=0x7f8f04100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:38.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.684+0000 7f8f09c29700 1 -- 172.21.15.142:0/1345914970 shutdown_connections 2024-04-26T21:26:38.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.684+0000 7f8f09c29700 1 -- 172.21.15.142:0/1345914970 wait complete. 2024-04-26T21:26:38.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.684+0000 7f8f09c29700 1 Processor -- start 2024-04-26T21:26:38.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.685+0000 7f8f09c29700 1 -- start start 2024-04-26T21:26:38.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.685+0000 7f8f09c29700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f0419ab00 0x7f8f0419af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:38.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.685+0000 7f8f09c29700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f0419b460 0x7f8f041a00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:38.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.685+0000 7f8f09c29700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f041a0630 0x7f8f041a2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:38.688 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.685+0000 7f8f09c29700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8f04108ef0 con 0x7f8f0419ab00 2024-04-26T21:26:38.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.685+0000 7f8f09c29700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8f04108d70 con 0x7f8f041a0630 2024-04-26T21:26:38.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.685+0000 7f8f09c29700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8f04109070 con 0x7f8f0419b460 2024-04-26T21:26:38.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.685+0000 7f8f02ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f0419b460 0x7f8f041a00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:38.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.685+0000 7f8f037fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f0419ab00 0x7f8f0419af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:38.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.685+0000 7f8f02ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f0419b460 0x7f8f041a00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:49350/0 (socket says 172.21.15.142:49350) 2024-04-26T21:26:38.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.685+0000 7f8f02ffd700 1 -- 172.21.15.142:0/2715989638 learned_addr learned my addr 172.21.15.142:0/2715989638 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-04-26T21:26:38.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.685+0000 7f8f037fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f0419ab00 0x7f8f0419af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.142:59722/0 (socket says 172.21.15.142:59722) 2024-04-26T21:26:38.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.685+0000 7f8f03fff700 1 --2- 172.21.15.142:0/2715989638 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f041a0630 0x7f8f041a2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:38.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.685+0000 7f8f02ffd700 1 -- 172.21.15.142:0/2715989638 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f041a0630 msgr2=0x7f8f041a2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:38.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.685+0000 7f8f02ffd700 1 --2- 172.21.15.142:0/2715989638 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f041a0630 0x7f8f041a2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:38.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.685+0000 7f8f02ffd700 1 -- 172.21.15.142:0/2715989638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f0419ab00 msgr2=0x7f8f0419af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:38.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.685+0000 7f8f02ffd700 1 --2- 172.21.15.142:0/2715989638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f0419ab00 0x7f8f0419af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:38.689 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.685+0000 7f8f02ffd700 1 -- 172.21.15.142:0/2715989638 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f041a3350 con 0x7f8f0419b460 2024-04-26T21:26:38.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.685+0000 7f8f037fe700 1 --2- 172.21.15.142:0/2715989638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f0419ab00 0x7f8f0419af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:26:38.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.685+0000 7f8f02ffd700 1 --2- 172.21.15.142:0/2715989638 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f0419b460 0x7f8f041a00f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800b780 tx=0x7f8ef800ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:38.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.686+0000 7f8f00ff9700 1 -- 172.21.15.142:0/2715989638 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8016070 con 0x7f8f0419b460 2024-04-26T21:26:38.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.686+0000 7f8f00ff9700 1 -- 172.21.15.142:0/2715989638 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ef80036a0 con 0x7f8f0419b460 2024-04-26T21:26:38.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.686+0000 7f8f09c29700 1 -- 172.21.15.142:0/2715989638 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f041a3640 con 0x7f8f0419b460 2024-04-26T21:26:38.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.686+0000 7f8f00ff9700 1 -- 172.21.15.142:0/2715989638 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef80119c0 con 0x7f8f0419b460 2024-04-26T21:26:38.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.686+0000 7f8f09c29700 1 -- 172.21.15.142:0/2715989638 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f041a3c50 con 0x7f8f0419b460 2024-04-26T21:26:38.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.686+0000 7f8f00ff9700 1 -- 172.21.15.142:0/2715989638 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f8ef8019070 con 0x7f8f0419b460 2024-04-26T21:26:38.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.687+0000 7f8f09c29700 1 -- 172.21.15.142:0/2715989638 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f04064a80 con 0x7f8f0419b460 2024-04-26T21:26:38.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.687+0000 7f8f00ff9700 1 --2- 172.21.15.142:0/2715989638 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8eec039ae0 0x7f8eec03bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:38.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.687+0000 7f8f00ff9700 1 -- 172.21.15.142:0/2715989638 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8ef804b310 con 0x7f8f0419b460 2024-04-26T21:26:38.690 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.687+0000 7f8f037fe700 1 --2- 172.21.15.142:0/2715989638 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8eec039ae0 0x7f8eec03bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:38.691 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.687+0000 7f8f037fe700 1 --2- 172.21.15.142:0/2715989638 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8eec039ae0 0x7f8eec03bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4000c00 tx=0x7f8ef401a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:38.692 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.692+0000 7f8f00ff9700 1 -- 172.21.15.142:0/2715989638 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8ef8003850 con 0x7f8f0419b460 2024-04-26T21:26:38.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:38 smithi142 ceph-mon[32944]: Reconfiguring mon.b (monmap changed)... 2024-04-26T21:26:38.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:38 smithi142 ceph-mon[32944]: Reconfiguring daemon mon.b on smithi050 2024-04-26T21:26:38.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:38 smithi050 ceph-mon[32753]: Reconfiguring mon.b (monmap changed)... 2024-04-26T21:26:38.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:38 smithi050 ceph-mon[32753]: Reconfiguring daemon mon.b on smithi050 2024-04-26T21:26:38.908 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.907+0000 7f8f09c29700 1 -- 172.21.15.142:0/2715989638 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi046=x;smithi050=y;smithi142=z", "target": ["mon-mgr", ""]}) -- 0x7f8f04075b10 con 0x7f8eec039ae0 2024-04-26T21:26:38.914 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.914+0000 7f8f00ff9700 1 -- 172.21.15.142:0/2715989638 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f8f04075b10 con 0x7f8eec039ae0 2024-04-26T21:26:38.914 INFO:teuthology.orchestra.run.smithi142.stdout:Scheduled mgr update... 2024-04-26T21:26:38.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.916+0000 7f8f09c29700 1 -- 172.21.15.142:0/2715989638 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8eec039ae0 msgr2=0x7f8eec03bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:38.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.916+0000 7f8f09c29700 1 --2- 172.21.15.142:0/2715989638 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8eec039ae0 0x7f8eec03bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4000c00 tx=0x7f8ef401a040 comp rx=0 tx=0).stop 2024-04-26T21:26:38.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.916+0000 7f8f09c29700 1 -- 172.21.15.142:0/2715989638 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f0419b460 msgr2=0x7f8f041a00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:38.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.916+0000 7f8f09c29700 1 --2- 172.21.15.142:0/2715989638 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f0419b460 0x7f8f041a00f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800b780 tx=0x7f8ef800ba90 comp rx=0 tx=0).stop 2024-04-26T21:26:38.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.916+0000 7f8f09c29700 1 -- 172.21.15.142:0/2715989638 shutdown_connections 2024-04-26T21:26:38.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.916+0000 7f8f09c29700 1 --2- 172.21.15.142:0/2715989638 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8eec039ae0 0x7f8eec03bfa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:38.917 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.916+0000 7f8f09c29700 1 --2- 172.21.15.142:0/2715989638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f0419ab00 0x7f8f0419af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:38.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.916+0000 7f8f09c29700 1 --2- 172.21.15.142:0/2715989638 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f041a0630 0x7f8f041a2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:38.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.916+0000 7f8f09c29700 1 --2- 172.21.15.142:0/2715989638 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8f0419b460 0x7f8f041a00f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:38.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.916+0000 7f8f09c29700 1 -- 172.21.15.142:0/2715989638 >> 172.21.15.142:0/2715989638 conn(0x7f8f040fe550 msgr2=0x7f8f04100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:38.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.916+0000 7f8f09c29700 1 -- 172.21.15.142:0/2715989638 shutdown_connections 2024-04-26T21:26:38.918 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:26:38.916+0000 7f8f09c29700 1 -- 172.21.15.142:0/2715989638 wait complete. 2024-04-26T21:26:39.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:38 smithi046 ceph-mon[29201]: Reconfiguring mon.b (monmap changed)... 2024-04-26T21:26:39.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:38 smithi046 ceph-mon[29201]: Reconfiguring daemon mon.b on smithi050 2024-04-26T21:26:39.516 DEBUG:teuthology.orchestra.run.smithi050:mgr.y> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mgr.y.service 2024-04-26T21:26:39.519 DEBUG:teuthology.orchestra.run.smithi142:mgr.z> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mgr.z.service 2024-04-26T21:26:39.559 INFO:tasks.cephadm:Deploying OSDs... 2024-04-26T21:26:39.559 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-26T21:26:39.560 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T21:26:39.579 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T21:26:39.579 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_1 2024-04-26T21:26:39.637 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T21:26:39.637 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T21:26:39.637 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-26T21:26:39.637 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T21:26:39.637 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T21:26:39.637 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-04-26 21:25:43.004253507 +0000 2024-04-26T21:26:39.637 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-04-26 21:20:48.758716121 +0000 2024-04-26T21:26:39.638 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-04-26 21:20:48.758716121 +0000 2024-04-26T21:26:39.638 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-04-26 21:20:48.758716121 +0000 2024-04-26T21:26:39.638 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T21:26:39.706 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-04-26T21:26:39.706 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-04-26T21:26:39.706 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.00013365 s, 3.8 MB/s 2024-04-26T21:26:39.707 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T21:26:39.766 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_2 2024-04-26T21:26:39.823 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T21:26:39.823 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T21:26:39.823 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-26T21:26:39.823 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T21:26:39.823 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T21:26:39.823 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-04-26 21:25:43.032253227 +0000 2024-04-26T21:26:39.823 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-04-26 21:20:49.092712784 +0000 2024-04-26T21:26:39.823 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-04-26 21:20:49.092712784 +0000 2024-04-26T21:26:39.823 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-04-26 21:20:49.092712784 +0000 2024-04-26T21:26:39.823 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T21:26:39.831 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:39 smithi142 ceph-mon[32944]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:39.831 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:39 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:39.831 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:39 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:39.831 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:39 smithi142 ceph-mon[32944]: Reconfiguring mon.c (monmap changed)... 2024-04-26T21:26:39.831 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:39 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T21:26:39.831 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:39 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T21:26:39.831 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:39 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:39.832 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:39 smithi142 ceph-mon[32944]: Reconfiguring daemon mon.c on smithi142 2024-04-26T21:26:39.832 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:39 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:39.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:39 smithi050 ceph-mon[32753]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:39.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:39 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:39.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:39 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:39.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:39 smithi050 ceph-mon[32753]: Reconfiguring mon.c (monmap changed)... 2024-04-26T21:26:39.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:39 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T21:26:39.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:39 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T21:26:39.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:39 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:39.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:39 smithi050 ceph-mon[32753]: Reconfiguring daemon mon.c on smithi142 2024-04-26T21:26:39.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:39 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:39.888 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-04-26T21:26:39.888 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-04-26T21:26:39.888 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000101499 s, 5.0 MB/s 2024-04-26T21:26:39.889 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T21:26:39.947 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_3 2024-04-26T21:26:40.004 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T21:26:40.004 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T21:26:40.004 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-26T21:26:40.004 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T21:26:40.004 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T21:26:40.004 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-04-26 21:25:43.059252956 +0000 2024-04-26T21:26:40.004 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-04-26 21:20:49.416709547 +0000 2024-04-26T21:26:40.004 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-04-26 21:20:49.416709547 +0000 2024-04-26T21:26:40.004 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-04-26 21:20:49.416709547 +0000 2024-04-26T21:26:40.005 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T21:26:40.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:39 smithi046 ceph-mon[29201]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:40.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:39 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:40.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:39 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:40.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:39 smithi046 ceph-mon[29201]: Reconfiguring mon.c (monmap changed)... 2024-04-26T21:26:40.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:39 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T21:26:40.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:39 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T21:26:40.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:39 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:40.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:39 smithi046 ceph-mon[29201]: Reconfiguring daemon mon.c on smithi142 2024-04-26T21:26:40.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:39 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:40.068 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-04-26T21:26:40.068 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-04-26T21:26:40.068 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000114128 s, 4.5 MB/s 2024-04-26T21:26:40.069 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T21:26:40.126 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_4 2024-04-26T21:26:40.182 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T21:26:40.182 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T21:26:40.182 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-26T21:26:40.182 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T21:26:40.183 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T21:26:40.183 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-04-26 21:25:43.087252676 +0000 2024-04-26T21:26:40.183 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-04-26 21:20:49.777705941 +0000 2024-04-26T21:26:40.183 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-04-26 21:20:49.777705941 +0000 2024-04-26T21:26:40.183 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-04-26 21:20:49.777705941 +0000 2024-04-26T21:26:40.183 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T21:26:40.245 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-04-26T21:26:40.246 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-04-26T21:26:40.246 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 9.6491e-05 s, 5.3 MB/s 2024-04-26T21:26:40.247 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T21:26:40.303 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T21:26:40.303 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T21:26:40.323 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T21:26:40.323 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_1 2024-04-26T21:26:40.381 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T21:26:40.382 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T21:26:40.382 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-26T21:26:40.382 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T21:26:40.382 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T21:26:40.382 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-04-26 21:26:26.211769774 +0000 2024-04-26T21:26:40.382 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-04-26 21:21:08.814060742 +0000 2024-04-26T21:26:40.382 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-04-26 21:21:08.814060742 +0000 2024-04-26T21:26:40.382 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-04-26 21:21:08.814060742 +0000 2024-04-26T21:26:40.383 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T21:26:40.449 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-04-26T21:26:40.450 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-04-26T21:26:40.450 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000117897 s, 4.3 MB/s 2024-04-26T21:26:40.451 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T21:26:40.511 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_2 2024-04-26T21:26:40.568 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T21:26:40.568 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T21:26:40.568 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-26T21:26:40.568 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T21:26:40.568 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T21:26:40.568 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-04-26 21:26:26.211769774 +0000 2024-04-26T21:26:40.569 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-04-26 21:21:09.156051740 +0000 2024-04-26T21:26:40.569 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-04-26 21:21:09.156051740 +0000 2024-04-26T21:26:40.569 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-04-26 21:21:09.156051740 +0000 2024-04-26T21:26:40.569 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T21:26:40.635 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-04-26T21:26:40.635 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-04-26T21:26:40.635 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000121803 s, 4.2 MB/s 2024-04-26T21:26:40.636 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T21:26:40.694 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_3 2024-04-26T21:26:40.751 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T21:26:40.752 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T21:26:40.752 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-26T21:26:40.752 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T21:26:40.752 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T21:26:40.752 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-04-26 21:26:26.212769749 +0000 2024-04-26T21:26:40.752 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-04-26 21:21:09.489042975 +0000 2024-04-26T21:26:40.752 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-04-26 21:21:09.489042975 +0000 2024-04-26T21:26:40.752 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-04-26 21:21:09.489042975 +0000 2024-04-26T21:26:40.752 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T21:26:40.814 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:40 smithi050 ceph-mon[32753]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi046=x;smithi050=y;smithi142=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:26:40.814 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:40 smithi050 ceph-mon[32753]: Saving service mgr spec with placement smithi046=x;smithi050=y;smithi142=z;count:3 2024-04-26T21:26:40.815 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:40 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:40.815 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:40 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:40.815 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:40 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:40.817 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-04-26T21:26:40.817 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-04-26T21:26:40.817 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000115238 s, 4.4 MB/s 2024-04-26T21:26:40.818 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T21:26:40.876 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_4 2024-04-26T21:26:40.933 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T21:26:40.933 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T21:26:40.933 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-26T21:26:40.933 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T21:26:40.934 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T21:26:40.934 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-04-26 21:26:26.212769749 +0000 2024-04-26T21:26:40.934 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-04-26 21:21:09.806034632 +0000 2024-04-26T21:26:40.934 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-04-26 21:21:09.806034632 +0000 2024-04-26T21:26:40.934 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-04-26 21:21:09.806034632 +0000 2024-04-26T21:26:40.934 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T21:26:40.998 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-04-26T21:26:40.998 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-04-26T21:26:40.998 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000119563 s, 4.3 MB/s 2024-04-26T21:26:41.000 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T21:26:41.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:40 smithi142 ceph-mon[32944]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi046=x;smithi050=y;smithi142=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:26:41.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:40 smithi142 ceph-mon[32944]: Saving service mgr spec with placement smithi046=x;smithi050=y;smithi142=z;count:3 2024-04-26T21:26:41.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:40 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:41.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:40 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:41.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:40 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:41.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:40 smithi046 ceph-mon[29201]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi046=x;smithi050=y;smithi142=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:26:41.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:40 smithi046 ceph-mon[29201]: Saving service mgr spec with placement smithi046=x;smithi050=y;smithi142=z;count:3 2024-04-26T21:26:41.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:40 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:41.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:40 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:41.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:40 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:41.056 DEBUG:teuthology.orchestra.run.smithi142:> set -ex 2024-04-26T21:26:41.056 DEBUG:teuthology.orchestra.run.smithi142:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T21:26:41.076 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T21:26:41.076 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_1 2024-04-26T21:26:41.133 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T21:26:41.133 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T21:26:41.133 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-26T21:26:41.133 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T21:26:41.133 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T21:26:41.133 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-04-26 21:26:16.068699291 +0000 2024-04-26T21:26:41.133 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-04-26 21:21:03.763929250 +0000 2024-04-26T21:26:41.133 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-04-26 21:21:03.763929250 +0000 2024-04-26T21:26:41.134 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: 2024-04-26 21:21:03.763929250 +0000 2024-04-26T21:26:41.134 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T21:26:41.198 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-04-26T21:26:41.198 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-04-26T21:26:41.198 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000273668 s, 1.9 MB/s 2024-04-26T21:26:41.199 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T21:26:41.255 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_2 2024-04-26T21:26:41.310 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T21:26:41.310 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T21:26:41.310 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-26T21:26:41.310 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T21:26:41.310 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T21:26:41.310 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-04-26 21:26:16.068699291 +0000 2024-04-26T21:26:41.310 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-04-26 21:21:04.120922361 +0000 2024-04-26T21:26:41.310 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-04-26 21:21:04.120922361 +0000 2024-04-26T21:26:41.310 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: 2024-04-26 21:21:04.120922361 +0000 2024-04-26T21:26:41.310 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T21:26:41.372 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-04-26T21:26:41.372 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-04-26T21:26:41.372 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000201658 s, 2.5 MB/s 2024-04-26T21:26:41.374 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T21:26:41.429 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_3 2024-04-26T21:26:41.483 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T21:26:41.483 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T21:26:41.483 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-26T21:26:41.483 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T21:26:41.483 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T21:26:41.483 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-04-26 21:26:16.068699291 +0000 2024-04-26T21:26:41.483 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-04-26 21:21:04.469915626 +0000 2024-04-26T21:26:41.483 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-04-26 21:21:04.469915626 +0000 2024-04-26T21:26:41.483 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: 2024-04-26 21:21:04.469915626 +0000 2024-04-26T21:26:41.484 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T21:26:41.547 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-04-26T21:26:41.547 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-04-26T21:26:41.547 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000186326 s, 2.7 MB/s 2024-04-26T21:26:41.548 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T21:26:41.604 DEBUG:teuthology.orchestra.run.smithi142:> stat /dev/vg_nvme/lv_4 2024-04-26T21:26:41.660 INFO:teuthology.orchestra.run.smithi142.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T21:26:41.660 INFO:teuthology.orchestra.run.smithi142.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T21:26:41.660 INFO:teuthology.orchestra.run.smithi142.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-26T21:26:41.660 INFO:teuthology.orchestra.run.smithi142.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T21:26:41.660 INFO:teuthology.orchestra.run.smithi142.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T21:26:41.660 INFO:teuthology.orchestra.run.smithi142.stdout:Access: 2024-04-26 21:26:16.069699273 +0000 2024-04-26T21:26:41.660 INFO:teuthology.orchestra.run.smithi142.stdout:Modify: 2024-04-26 21:21:04.808909085 +0000 2024-04-26T21:26:41.660 INFO:teuthology.orchestra.run.smithi142.stdout:Change: 2024-04-26 21:21:04.808909085 +0000 2024-04-26T21:26:41.660 INFO:teuthology.orchestra.run.smithi142.stdout: Birth: 2024-04-26 21:21:04.808909085 +0000 2024-04-26T21:26:41.661 DEBUG:teuthology.orchestra.run.smithi142:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T21:26:41.724 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records in 2024-04-26T21:26:41.724 INFO:teuthology.orchestra.run.smithi142.stderr:1+0 records out 2024-04-26T21:26:41.724 INFO:teuthology.orchestra.run.smithi142.stderr:512 bytes copied, 0.000206901 s, 2.5 MB/s 2024-04-26T21:26:41.725 DEBUG:teuthology.orchestra.run.smithi142:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T21:26:41.781 INFO:tasks.cephadm:Deploying osd.0 on smithi046 with /dev/vg_nvme/lv_4... 2024-04-26T21:26:41.781 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T21:26:41.922 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:41 smithi142 ceph-mon[32944]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:41.922 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:41 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:41.923 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:41 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:41.923 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:41 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:41.923 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:41 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T21:26:41.923 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:41 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T21:26:41.923 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:41 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T21:26:41.923 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:41 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:42.031 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:26:42.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:41 smithi046 ceph-mon[29201]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:42.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:41 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:42.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:41 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:42.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:41 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:42.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:41 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T21:26:42.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:41 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T21:26:42.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:41 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T21:26:42.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:41 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:42.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:41 smithi050 ceph-mon[32753]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:42.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:41 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:42.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:41 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:42.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:41 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:42.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:41 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T21:26:42.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:41 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T21:26:42.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:41 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T21:26:42.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:41 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:42.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:42 smithi046 ceph-mon[29201]: Deploying daemon mgr.z on smithi142 2024-04-26T21:26:42.980 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:42 smithi142 ceph-mon[32944]: Deploying daemon mgr.z on smithi142 2024-04-26T21:26:43.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:42 smithi050 ceph-mon[32753]: Deploying daemon mgr.z on smithi142 2024-04-26T21:26:43.291 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:43 smithi142 systemd[1]: Starting Ceph mgr.z for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:26:43.675 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:43 smithi142 podman[34669]: 2024-04-26 21:26:43.396515369 +0000 UTC m=+0.110195773 container create 7662cfb726a521b9d91a41256b386ed3cd4e06b42274b826c4af2b52b4f24a95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-26T21:26:43.675 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:43 smithi142 podman[34669]: 2024-04-26 21:26:43.304489702 +0000 UTC m=+0.018170067 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:26:43.675 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:43 smithi142 podman[34669]: 2024-04-26 21:26:43.590503196 +0000 UTC m=+0.304183555 container init 7662cfb726a521b9d91a41256b386ed3cd4e06b42274b826c4af2b52b4f24a95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-26T21:26:43.675 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:43 smithi142 podman[34669]: 2024-04-26 21:26:43.592976011 +0000 UTC m=+0.306656373 container start 7662cfb726a521b9d91a41256b386ed3cd4e06b42274b826c4af2b52b4f24a95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, ceph=True) 2024-04-26T21:26:43.675 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:43 smithi142 ceph-mgr[34688]: -- 172.21.15.142:0/1001237094 <== mon.1 v2:172.21.15.142:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5648135d25a0 con 0x5648135a7000 2024-04-26T21:26:43.932 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:43 smithi142 ceph-mon[32944]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:43.932 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:43 smithi142 bash[34669]: 7662cfb726a521b9d91a41256b386ed3cd4e06b42274b826c4af2b52b4f24a95 2024-04-26T21:26:43.932 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:43 smithi142 systemd[1]: Started Ceph mgr.z for 36c7013a-0413-11ef-bc93-c7b262605968. 2024-04-26T21:26:43.932 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:43 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z[34684]: 2024-04-26T21:26:43.746+0000 7f06405a9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T21:26:43.932 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:43 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z[34684]: 2024-04-26T21:26:43.931+0000 7f06405a9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T21:26:43.946 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T21:26:43.946 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T21:26:43.946 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2024-04-26T21:26:43.946 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2024-04-26T21:26:43.946 INFO:teuthology.orchestra.run.smithi046.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00822673 s, 1.3 GB/s 2024-04-26T21:26:43.946 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2024-04-26T21:26:43.946 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:43 smithi046 ceph-mon[29201]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:44.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:43 smithi050 ceph-mon[32753]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:44.290 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:44 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z[34684]: 2024-04-26T21:26:44.032+0000 7f06405a9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T21:26:44.626 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi046:vg_nvme/lv_4 2024-04-26T21:26:44.750 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:44 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z[34684]: 2024-04-26T21:26:44.337+0000 7f06405a9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T21:26:44.750 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:44 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z[34684]: 2024-04-26T21:26:44.420+0000 7f06405a9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T21:26:44.879 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:26:45.001 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:44 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:45.001 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:44 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:45.001 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:44 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:45.001 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:44 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T21:26:45.001 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:44 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T21:26:45.001 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:44 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T21:26:45.001 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:44 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:45.001 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:44 smithi046 ceph-mon[29201]: Deploying daemon mgr.y on smithi050 2024-04-26T21:26:45.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:44 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:45.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:44 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:45.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:44 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:45.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:44 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T21:26:45.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:44 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T21:26:45.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:44 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T21:26:45.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:44 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:45.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:44 smithi142 ceph-mon[32944]: Deploying daemon mgr.y on smithi050 2024-04-26T21:26:45.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:44 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:45.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:44 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:45.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:44 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:45.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:44 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T21:26:45.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:44 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T21:26:45.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:44 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T21:26:45.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:44 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:45.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:44 smithi050 ceph-mon[32753]: Deploying daemon mgr.y on smithi050 2024-04-26T21:26:45.947 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:45 smithi142 ceph-mon[32944]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:45.947 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:45 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z[34684]: 2024-04-26T21:26:45.605+0000 7f06405a9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T21:26:45.947 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:45 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z[34684]: 2024-04-26T21:26:45.772+0000 7f06405a9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T21:26:45.947 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:45 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z[34684]: 2024-04-26T21:26:45.946+0000 7f06405a9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T21:26:46.040 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:45 smithi050 ceph-mon[32753]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:46.040 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:45 smithi050 systemd[1]: Starting Ceph mgr.y for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:26:46.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:45 smithi046 ceph-mon[29201]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:46.300 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:46 smithi050 podman[34130]: 2024-04-26 21:26:45.942617015 +0000 UTC m=+0.018305446 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:26:46.300 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:46 smithi050 podman[34130]: 2024-04-26 21:26:46.181700298 +0000 UTC m=+0.257388717 container create 9a30793b0e455a581f2249bd613f3489e6ef15893d371f4594131a0b871b5d3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-26T21:26:46.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.326+0000 7f9660c36700 1 -- 172.21.15.46:0/1752589582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f965c103f30 msgr2=0x7f965c106320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:46.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.326+0000 7f9660c36700 1 --2- 172.21.15.46:0/1752589582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f965c103f30 0x7f965c106320 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f964c0098f0 tx=0x7f964c009c00 comp rx=0 tx=0).stop 2024-04-26T21:26:46.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.326+0000 7f9660c36700 1 -- 172.21.15.46:0/1752589582 shutdown_connections 2024-04-26T21:26:46.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.326+0000 7f9660c36700 1 --2- 172.21.15.46:0/1752589582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f965c103f30 0x7f965c106320 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:46.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.326+0000 7f9660c36700 1 --2- 172.21.15.46:0/1752589582 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f965c109190 0x7f965c10b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:46.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.326+0000 7f9660c36700 1 --2- 172.21.15.46:0/1752589582 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f965c106860 0x7f965c108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:46.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.326+0000 7f9660c36700 1 -- 172.21.15.46:0/1752589582 >> 172.21.15.46:0/1752589582 conn(0x7f965c0fd940 msgr2=0x7f965c0ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:46.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.326+0000 7f9660c36700 1 -- 172.21.15.46:0/1752589582 shutdown_connections 2024-04-26T21:26:46.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.326+0000 7f9660c36700 1 -- 172.21.15.46:0/1752589582 wait complete. 2024-04-26T21:26:46.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.327+0000 7f9660c36700 1 Processor -- start 2024-04-26T21:26:46.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.327+0000 7f9660c36700 1 -- start start 2024-04-26T21:26:46.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.327+0000 7f9660c36700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f965c103f30 0x7f965c103890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:46.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.327+0000 7f9660c36700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f965c106860 0x7f965c101ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:46.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.327+0000 7f9660c36700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f965c109190 0x7f965c102420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:46.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.327+0000 7f9660c36700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f965c10ef60 con 0x7f965c109190 2024-04-26T21:26:46.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.327+0000 7f9660c36700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f965c10ede0 con 0x7f965c103f30 2024-04-26T21:26:46.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.327+0000 7f9660c36700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f965c10f0e0 con 0x7f965c106860 2024-04-26T21:26:46.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.327+0000 7f965a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f965c103f30 0x7f965c103890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:46.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.327+0000 7f965ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f965c109190 0x7f965c102420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:46.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.327+0000 7f965a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f965c103f30 0x7f965c103890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.46:41858/0 (socket says 172.21.15.46:41858) 2024-04-26T21:26:46.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.327+0000 7f965a59c700 1 -- 172.21.15.46:0/963551805 learned_addr learned my addr 172.21.15.46:0/963551805 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:26:46.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.327+0000 7f9659d9b700 1 --2- 172.21.15.46:0/963551805 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f965c106860 0x7f965c101ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:46.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.327+0000 7f965a59c700 1 -- 172.21.15.46:0/963551805 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f965c106860 msgr2=0x7f965c101ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:46.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.327+0000 7f965a59c700 1 --2- 172.21.15.46:0/963551805 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f965c106860 0x7f965c101ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:46.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.327+0000 7f965a59c700 1 -- 172.21.15.46:0/963551805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f965c109190 msgr2=0x7f965c102420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:46.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.327+0000 7f965a59c700 1 --2- 172.21.15.46:0/963551805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f965c109190 0x7f965c102420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:46.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.327+0000 7f965a59c700 1 -- 172.21.15.46:0/963551805 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f965c102cd0 con 0x7f965c103f30 2024-04-26T21:26:46.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.328+0000 7f965a59c700 1 --2- 172.21.15.46:0/963551805 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f965c103f30 0x7f965c103890 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f964c00fc20 tx=0x7f964c005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:46.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.328+0000 7f964b7fe700 1 -- 172.21.15.46:0/963551805 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c005490 con 0x7f965c103f30 2024-04-26T21:26:46.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.328+0000 7f9660c36700 1 -- 172.21.15.46:0/963551805 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f965c1a3710 con 0x7f965c103f30 2024-04-26T21:26:46.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.328+0000 7f964b7fe700 1 -- 172.21.15.46:0/963551805 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f964c005630 con 0x7f965c103f30 2024-04-26T21:26:46.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.328+0000 7f964b7fe700 1 -- 172.21.15.46:0/963551805 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c003b40 con 0x7f965c103f30 2024-04-26T21:26:46.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.328+0000 7f9660c36700 1 -- 172.21.15.46:0/963551805 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f965c1a3b10 con 0x7f965c103f30 2024-04-26T21:26:46.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.329+0000 7f964b7fe700 1 -- 172.21.15.46:0/963551805 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f964c003ce0 con 0x7f965c103f30 2024-04-26T21:26:46.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.329+0000 7f9660c36700 1 -- 172.21.15.46:0/963551805 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f965c06a110 con 0x7f965c103f30 2024-04-26T21:26:46.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.329+0000 7f964b7fe700 1 --2- 172.21.15.46:0/963551805 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f9644039890 0x7f964403bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:46.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.329+0000 7f964b7fe700 1 -- 172.21.15.46:0/963551805 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f964c04dac0 con 0x7f965c103f30 2024-04-26T21:26:46.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.329+0000 7f9659d9b700 1 --2- 172.21.15.46:0/963551805 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f9644039890 0x7f964403bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:46.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.329+0000 7f9659d9b700 1 --2- 172.21.15.46:0/963551805 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f9644039890 0x7f964403bd50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9650005fd0 tx=0x7f9650009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:46.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.334+0000 7f964b7fe700 1 -- 172.21.15.46:0/963551805 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f964c013030 con 0x7f965c103f30 2024-04-26T21:26:46.530 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:46 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z[34684]: 2024-04-26T21:26:46.231+0000 7f06405a9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T21:26:46.530 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:46 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z[34684]: 2024-04-26T21:26:46.323+0000 7f06405a9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T21:26:46.530 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:46 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z[34684]: 2024-04-26T21:26:46.528+0000 7f06405a9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T21:26:46.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:46.554+0000 7f9660c36700 1 -- 172.21.15.46:0/963551805 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f965c063860 con 0x7f9644039890 2024-04-26T21:26:46.618 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:46 smithi050 podman[34130]: 2024-04-26 21:26:46.367355379 +0000 UTC m=+0.443043799 container init 9a30793b0e455a581f2249bd613f3489e6ef15893d371f4594131a0b871b5d3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T21:26:46.618 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:46 smithi050 podman[34130]: 2024-04-26 21:26:46.370101572 +0000 UTC m=+0.445789992 container start 9a30793b0e455a581f2249bd613f3489e6ef15893d371f4594131a0b871b5d3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-26T21:26:46.618 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:46 smithi050 ceph-mgr[34149]: -- 172.21.15.50:0/1656975481 <== mon.2 v2:172.21.15.50:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56301ccda5a0 con 0x56301ccaf000 2024-04-26T21:26:46.618 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:46 smithi050 bash[34130]: 9a30793b0e455a581f2249bd613f3489e6ef15893d371f4594131a0b871b5d3a 2024-04-26T21:26:46.619 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:46 smithi050 systemd[1]: Started Ceph mgr.y for 36c7013a-0413-11ef-bc93-c7b262605968. 2024-04-26T21:26:46.619 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:46 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y[34145]: 2024-04-26T21:26:46.526+0000 7f9828096200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T21:26:46.790 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:46 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z[34684]: 2024-04-26T21:26:46.721+0000 7f06405a9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T21:26:47.118 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:46 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y[34145]: 2024-04-26T21:26:46.719+0000 7f9828096200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T21:26:47.118 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:46 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y[34145]: 2024-04-26T21:26:46.827+0000 7f9828096200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T21:26:47.504 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:47 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y[34145]: 2024-04-26T21:26:47.158+0000 7f9828096200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T21:26:47.505 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:47 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y[34145]: 2024-04-26T21:26:47.250+0000 7f9828096200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T21:26:47.738 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:47 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z[34684]: 2024-04-26T21:26:47.368+0000 7f06405a9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T21:26:47.741 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:47 smithi142 ceph-mon[32944]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:47.741 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:47 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:47.741 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:47 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:47.741 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:47 smithi142 ceph-mon[32944]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:26:47.741 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:47 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:47.741 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:47 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:26:47.741 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:47 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:26:47.741 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:47 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:47.741 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:47 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:47.741 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:47 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:47.794 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:47 smithi050 ceph-mon[32753]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:47.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:47 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:47.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:47 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:47.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:47 smithi050 ceph-mon[32753]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:26:47.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:47 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:47.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:47 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:26:47.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:47 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:26:47.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:47 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:47.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:47 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:47.795 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:47 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:47.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:47 smithi046 ceph-mon[29201]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:47.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:47 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:47.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:47 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:47.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:47 smithi046 ceph-mon[29201]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:26:47.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:47 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:47.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:47 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:26:47.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:47 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:26:47.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:47 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:47.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:47 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:47.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:47 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:48.041 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:47 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z[34684]: 2024-04-26T21:26:47.811+0000 7f06405a9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T21:26:48.041 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:47 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z[34684]: 2024-04-26T21:26:47.909+0000 7f06405a9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T21:26:48.541 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:48 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z[34684]: 2024-04-26T21:26:48.109+0000 7f06405a9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T21:26:48.868 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:48 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y[34145]: 2024-04-26T21:26:48.552+0000 7f9828096200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T21:26:48.868 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:48 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y[34145]: 2024-04-26T21:26:48.727+0000 7f9828096200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T21:26:48.975 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:48 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z[34684]: 2024-04-26T21:26:48.644+0000 7f06405a9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T21:26:48.976 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:48 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z[34684]: 2024-04-26T21:26:48.735+0000 7f06405a9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T21:26:49.177 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:48 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y[34145]: 2024-04-26T21:26:48.903+0000 7f9828096200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T21:26:49.293 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:48 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z[34684]: 2024-04-26T21:26:48.975+0000 7f06405a9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T21:26:49.293 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:49 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z[34684]: 2024-04-26T21:26:49.108+0000 7f06405a9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T21:26:49.294 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:49 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z[34684]: 2024-04-26T21:26:49.203+0000 7f06405a9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T21:26:49.294 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:49 smithi142 ceph-mon[32944]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:49.294 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:49 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:49.294 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:49 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:49.294 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:49 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:49.294 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:49 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:49.294 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:49 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:49.294 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:49 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:49.294 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:49 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:49.294 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:49 smithi142 ceph-mon[32944]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T21:26:49.294 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:49 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T21:26:49.294 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:49 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T21:26:49.295 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:49 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:49.295 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:49 smithi142 ceph-mon[32944]: Reconfiguring daemon mgr.x on smithi046 2024-04-26T21:26:49.507 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:49 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y[34145]: 2024-04-26T21:26:49.200+0000 7f9828096200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T21:26:49.508 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:49 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y[34145]: 2024-04-26T21:26:49.296+0000 7f9828096200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T21:26:49.508 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:49 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y[34145]: 2024-04-26T21:26:49.506+0000 7f9828096200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T21:26:49.508 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:49 smithi050 ceph-mon[32753]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:49.508 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:49 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:49.508 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:49 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:49.508 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:49 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:49.508 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:49 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:49.508 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:49 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:49.508 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:49 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:49.508 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:49 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:49.508 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:49 smithi050 ceph-mon[32753]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T21:26:49.508 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:49 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T21:26:49.509 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:49 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T21:26:49.509 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:49 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:49.509 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:49 smithi050 ceph-mon[32753]: Reconfiguring daemon mgr.x on smithi046 2024-04-26T21:26:49.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:49 smithi046 ceph-mon[29201]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:49.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:49 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:49.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:49 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:49.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:49 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:49.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:49 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:49.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:49 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:49.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:49 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:49.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:49 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:49.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:49 smithi046 ceph-mon[29201]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T21:26:49.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:49 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T21:26:49.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:49 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T21:26:49.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:49 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:49.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:49 smithi046 ceph-mon[29201]: Reconfiguring daemon mgr.x on smithi046 2024-04-26T21:26:49.620 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:49 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z[34684]: 2024-04-26T21:26:49.317+0000 7f06405a9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T21:26:49.621 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:49 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z[34684]: 2024-04-26T21:26:49.619+0000 7f06405a9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T21:26:49.868 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:49 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y[34145]: 2024-04-26T21:26:49.706+0000 7f9828096200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T21:26:50.541 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:50 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z[34684]: 2024-04-26T21:26:50.213+0000 7f06405a9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T21:26:50.541 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:26:50 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z[34684]: 2024-04-26T21:26:50.425+0000 7f06405a9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T21:26:50.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:50 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3988651116' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f53c8702-e4da-435f-a9dc-3911d0d36ef5"}]: dispatch 2024-04-26T21:26:50.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:50 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3988651116' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f53c8702-e4da-435f-a9dc-3911d0d36ef5"}]': finished 2024-04-26T21:26:50.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:50 smithi142 ceph-mon[32944]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T21:26:50.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:50 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:26:50.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:50 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:50.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:50 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3532514838' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:26:50.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:50 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3988651116' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f53c8702-e4da-435f-a9dc-3911d0d36ef5"}]: dispatch 2024-04-26T21:26:50.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:50 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3988651116' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f53c8702-e4da-435f-a9dc-3911d0d36ef5"}]': finished 2024-04-26T21:26:50.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:50 smithi046 ceph-mon[29201]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T21:26:50.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:50 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:26:50.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:50 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:50.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:50 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3532514838' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:26:50.618 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:50 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y[34145]: 2024-04-26T21:26:50.331+0000 7f9828096200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T21:26:50.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:50 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3988651116' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f53c8702-e4da-435f-a9dc-3911d0d36ef5"}]: dispatch 2024-04-26T21:26:50.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:50 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3988651116' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f53c8702-e4da-435f-a9dc-3911d0d36ef5"}]': finished 2024-04-26T21:26:50.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:50 smithi050 ceph-mon[32753]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T21:26:50.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:50 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:26:50.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:50 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:50.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:50 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3532514838' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:26:51.073 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:50 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y[34145]: 2024-04-26T21:26:50.764+0000 7f9828096200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T21:26:51.073 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:50 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y[34145]: 2024-04-26T21:26:50.864+0000 7f9828096200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T21:26:51.073 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:51 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y[34145]: 2024-04-26T21:26:51.070+0000 7f9828096200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T21:26:51.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:51.186+0000 7f964b7fe700 1 -- 172.21.15.46:0/963551805 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7f964c02caa0 con 0x7f965c103f30 2024-04-26T21:26:51.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:51 smithi050 ceph-mon[32753]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:51.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:51 smithi050 ceph-mon[32753]: Standby manager daemon z started 2024-04-26T21:26:51.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:51 smithi050 ceph-mon[32753]: from='mgr.? 172.21.15.142:0/3637472718' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T21:26:51.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:51 smithi050 ceph-mon[32753]: from='mgr.? 172.21.15.142:0/3637472718' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T21:26:51.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:51 smithi050 ceph-mon[32753]: from='mgr.? 172.21.15.142:0/3637472718' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T21:26:51.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:51 smithi050 ceph-mon[32753]: from='mgr.? 172.21.15.142:0/3637472718' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T21:26:51.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:51 smithi142 ceph-mon[32944]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:51.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:51 smithi142 ceph-mon[32944]: Standby manager daemon z started 2024-04-26T21:26:51.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:51 smithi142 ceph-mon[32944]: from='mgr.? 172.21.15.142:0/3637472718' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T21:26:51.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:51 smithi142 ceph-mon[32944]: from='mgr.? 172.21.15.142:0/3637472718' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T21:26:51.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:51 smithi142 ceph-mon[32944]: from='mgr.? 172.21.15.142:0/3637472718' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T21:26:51.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:51 smithi142 ceph-mon[32944]: from='mgr.? 172.21.15.142:0/3637472718' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T21:26:51.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:51 smithi046 ceph-mon[29201]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:51.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:51 smithi046 ceph-mon[29201]: Standby manager daemon z started 2024-04-26T21:26:51.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:51 smithi046 ceph-mon[29201]: from='mgr.? 172.21.15.142:0/3637472718' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T21:26:51.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:51 smithi046 ceph-mon[29201]: from='mgr.? 172.21.15.142:0/3637472718' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T21:26:51.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:51 smithi046 ceph-mon[29201]: from='mgr.? 172.21.15.142:0/3637472718' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T21:26:51.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:51 smithi046 ceph-mon[29201]: from='mgr.? 172.21.15.142:0/3637472718' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T21:26:51.868 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:51 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y[34145]: 2024-04-26T21:26:51.613+0000 7f9828096200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T21:26:51.868 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:51 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y[34145]: 2024-04-26T21:26:51.707+0000 7f9828096200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T21:26:52.319 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:52 smithi050 ceph-mon[32753]: mgrmap e14: x(active, since 81s), standbys: z 2024-04-26T21:26:52.320 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:52 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T21:26:52.320 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:51 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y[34145]: 2024-04-26T21:26:51.962+0000 7f9828096200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T21:26:52.320 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:52 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y[34145]: 2024-04-26T21:26:52.100+0000 7f9828096200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T21:26:52.320 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:52 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y[34145]: 2024-04-26T21:26:52.199+0000 7f9828096200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T21:26:52.320 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:52 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y[34145]: 2024-04-26T21:26:52.318+0000 7f9828096200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T21:26:52.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:52 smithi142 ceph-mon[32944]: mgrmap e14: x(active, since 81s), standbys: z 2024-04-26T21:26:52.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:52 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T21:26:52.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:52 smithi046 ceph-mon[29201]: mgrmap e14: x(active, since 81s), standbys: z 2024-04-26T21:26:52.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:52 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T21:26:53.118 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:52 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y[34145]: 2024-04-26T21:26:52.634+0000 7f9828096200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T21:26:53.455 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:53 smithi046 ceph-mon[29201]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:53.486 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:53 smithi050 ceph-mon[32753]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:53.487 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:53 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y[34145]: 2024-04-26T21:26:53.261+0000 7f9828096200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T21:26:53.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:53 smithi142 ceph-mon[32944]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:53.868 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:26:53 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y[34145]: 2024-04-26T21:26:53.485+0000 7f9828096200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T21:26:54.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:26:54.200+0000 7f964b7fe700 1 -- 172.21.15.46:0/963551805 <== mon.2 v2:172.21.15.50:3300/0 8 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f964c00e3e0 con 0x7f965c103f30 2024-04-26T21:26:54.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:54 smithi142 ceph-mon[32944]: Standby manager daemon y started 2024-04-26T21:26:54.546 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:54 smithi142 ceph-mon[32944]: from='mgr.? 172.21.15.50:0/1410130588' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T21:26:54.546 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:54 smithi142 ceph-mon[32944]: from='mgr.? 172.21.15.50:0/1410130588' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T21:26:54.546 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:54 smithi142 ceph-mon[32944]: from='mgr.? 172.21.15.50:0/1410130588' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T21:26:54.546 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:54 smithi142 ceph-mon[32944]: from='mgr.? 172.21.15.50:0/1410130588' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T21:26:54.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:54 smithi046 ceph-mon[29201]: Standby manager daemon y started 2024-04-26T21:26:54.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:54 smithi046 ceph-mon[29201]: from='mgr.? 172.21.15.50:0/1410130588' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T21:26:54.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:54 smithi046 ceph-mon[29201]: from='mgr.? 172.21.15.50:0/1410130588' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T21:26:54.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:54 smithi046 ceph-mon[29201]: from='mgr.? 172.21.15.50:0/1410130588' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T21:26:54.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:54 smithi046 ceph-mon[29201]: from='mgr.? 172.21.15.50:0/1410130588' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T21:26:54.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:54 smithi050 ceph-mon[32753]: Standby manager daemon y started 2024-04-26T21:26:54.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:54 smithi050 ceph-mon[32753]: from='mgr.? 172.21.15.50:0/1410130588' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T21:26:54.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:54 smithi050 ceph-mon[32753]: from='mgr.? 172.21.15.50:0/1410130588' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T21:26:54.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:54 smithi050 ceph-mon[32753]: from='mgr.? 172.21.15.50:0/1410130588' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T21:26:54.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:54 smithi050 ceph-mon[32753]: from='mgr.? 172.21.15.50:0/1410130588' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T21:26:55.488 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:55 smithi046 ceph-mon[29201]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:55.488 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:55 smithi046 ceph-mon[29201]: mgrmap e15: x(active, since 84s), standbys: z, y 2024-04-26T21:26:55.488 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:55 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T21:26:55.488 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:55 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:55.488 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:55 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:55.488 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:55 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:55.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:55 smithi142 ceph-mon[32944]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:55.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:55 smithi142 ceph-mon[32944]: mgrmap e15: x(active, since 84s), standbys: z, y 2024-04-26T21:26:55.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:55 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T21:26:55.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:55 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:55.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:55 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:55.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:55 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:55.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:55 smithi050 ceph-mon[32753]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:55.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:55 smithi050 ceph-mon[32753]: mgrmap e15: x(active, since 84s), standbys: z, y 2024-04-26T21:26:55.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:55 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T21:26:55.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:55 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:55.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:55 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:55.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:55 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:56.200 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:56 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:56.200 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:56 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:56.200 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:56 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:56.389 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:56 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:56.390 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:56 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:56.390 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:56 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:56.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:56 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:56.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:56 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:56.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:56 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:26:58.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:57 smithi142 ceph-mon[32944]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:58.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:57 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T21:26:58.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:57 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:58.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:57 smithi046 ceph-mon[29201]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:58.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:57 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T21:26:58.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:57 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:58.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:57 smithi050 ceph-mon[32753]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:26:58.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:57 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T21:26:58.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:57 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:58.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:58 smithi046 ceph-mon[29201]: Deploying daemon osd.0 on smithi046 2024-04-26T21:26:59.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:58 smithi142 ceph-mon[32944]: Deploying daemon osd.0 on smithi046 2024-04-26T21:26:59.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:58 smithi050 ceph-mon[32753]: Deploying daemon osd.0 on smithi046 2024-04-26T21:27:00.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:26:59 smithi142 ceph-mon[32944]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:27:00.042 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:26:59 smithi046 ceph-mon[29201]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:27:00.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:26:59 smithi050 ceph-mon[32753]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:27:02.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:01 smithi142 ceph-mon[32944]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:27:02.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:01 smithi046 ceph-mon[29201]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:27:02.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:01 smithi050 ceph-mon[32753]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:27:03.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:03 smithi046 ceph-mon[29201]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:27:03.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:03 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:03.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:03 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:03.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:03 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:27:04.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:03 smithi142 ceph-mon[32944]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:27:04.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:03 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:04.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:03 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:04.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:03 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:27:04.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:03 smithi050 ceph-mon[32753]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:27:04.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:03 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:04.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:03 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:04.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:03 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:27:06.027 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:05 smithi046 ceph-mon[29201]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:27:06.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:05 smithi142 ceph-mon[32944]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:27:06.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:05 smithi050 ceph-mon[32753]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:27:07.165 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:07 smithi046 ceph-mon[29201]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:27:07.165 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:07 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:07.165 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:07 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:07.165 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:07 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:07.165 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:07 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:07.166 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:07 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:07.166 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:07 smithi046 ceph-mon[29201]: from='osd.0 [v2:172.21.15.46:6802/2762665125,v1:172.21.15.46:6803/2762665125]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T21:27:07.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:27:07.242+0000 7f964b7fe700 1 -- 172.21.15.46:0/963551805 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f965c063860 con 0x7f9644039890 2024-04-26T21:27:07.243 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 0 on host 'smithi046' 2024-04-26T21:27:07.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:27:07.246+0000 7f9660c36700 1 -- 172.21.15.46:0/963551805 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f9644039890 msgr2=0x7f964403bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:07.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:27:07.246+0000 7f9660c36700 1 --2- 172.21.15.46:0/963551805 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f9644039890 0x7f964403bd50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9650005fd0 tx=0x7f9650009310 comp rx=0 tx=0).stop 2024-04-26T21:27:07.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:27:07.246+0000 7f9660c36700 1 -- 172.21.15.46:0/963551805 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f965c103f30 msgr2=0x7f965c103890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:07.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:27:07.246+0000 7f9660c36700 1 --2- 172.21.15.46:0/963551805 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f965c103f30 0x7f965c103890 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f964c00fc20 tx=0x7f964c005270 comp rx=0 tx=0).stop 2024-04-26T21:27:07.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:27:07.246+0000 7f9660c36700 1 -- 172.21.15.46:0/963551805 shutdown_connections 2024-04-26T21:27:07.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:27:07.246+0000 7f9660c36700 1 --2- 172.21.15.46:0/963551805 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f9644039890 0x7f964403bd50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:07.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:27:07.246+0000 7f9660c36700 1 --2- 172.21.15.46:0/963551805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f965c109190 0x7f965c102420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:07.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:27:07.246+0000 7f9660c36700 1 --2- 172.21.15.46:0/963551805 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f965c103f30 0x7f965c103890 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:07.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:27:07.246+0000 7f9660c36700 1 --2- 172.21.15.46:0/963551805 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f965c106860 0x7f965c101ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:07.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:27:07.246+0000 7f9660c36700 1 -- 172.21.15.46:0/963551805 >> 172.21.15.46:0/963551805 conn(0x7f965c0fd940 msgr2=0x7f965c105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:07.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:27:07.246+0000 7f9660c36700 1 -- 172.21.15.46:0/963551805 shutdown_connections 2024-04-26T21:27:07.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:27:07.246+0000 7f9660c36700 1 -- 172.21.15.46:0/963551805 wait complete. 2024-04-26T21:27:07.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:07 smithi050 ceph-mon[32753]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:27:07.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:07 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:07.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:07 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:07.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:07 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:07.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:07 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:07.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:07 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:07.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:07 smithi050 ceph-mon[32753]: from='osd.0 [v2:172.21.15.46:6802/2762665125,v1:172.21.15.46:6803/2762665125]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T21:27:07.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:07 smithi142 ceph-mon[32944]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:27:07.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:07 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:07.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:07 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:07.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:07 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:07.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:07 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:07.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:07 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:07.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:07 smithi142 ceph-mon[32944]: from='osd.0 [v2:172.21.15.46:6802/2762665125,v1:172.21.15.46:6803/2762665125]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T21:27:07.948 DEBUG:teuthology.orchestra.run.smithi046:osd.0> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.0.service 2024-04-26T21:27:07.950 INFO:tasks.cephadm:Deploying osd.1 on smithi050 with /dev/vg_nvme/lv_4... 2024-04-26T21:27:07.950 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T21:27:08.203 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T21:27:08.307 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:08 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:08.307 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:08 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:08.308 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:08 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:27:08.308 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:08 smithi050 ceph-mon[32753]: from='osd.0 [v2:172.21.15.46:6802/2762665125,v1:172.21.15.46:6803/2762665125]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T21:27:08.308 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:08 smithi050 ceph-mon[32753]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T21:27:08.308 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:08 smithi050 ceph-mon[32753]: from='osd.0 [v2:172.21.15.46:6802/2762665125,v1:172.21.15.46:6803/2762665125]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-26T21:27:08.308 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:08 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:27:08.452 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:08 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:08.452 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:08 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:08.452 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:08 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:27:08.452 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:08 smithi046 ceph-mon[29201]: from='osd.0 [v2:172.21.15.46:6802/2762665125,v1:172.21.15.46:6803/2762665125]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T21:27:08.452 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:08 smithi046 ceph-mon[29201]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T21:27:08.452 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:08 smithi046 ceph-mon[29201]: from='osd.0 [v2:172.21.15.46:6802/2762665125,v1:172.21.15.46:6803/2762665125]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-26T21:27:08.452 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:08 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:27:08.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:08 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:08.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:08 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:08.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:08 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:27:08.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:08 smithi142 ceph-mon[32944]: from='osd.0 [v2:172.21.15.46:6802/2762665125,v1:172.21.15.46:6803/2762665125]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T21:27:08.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:08 smithi142 ceph-mon[32944]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T21:27:08.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:08 smithi142 ceph-mon[32944]: from='osd.0 [v2:172.21.15.46:6802/2762665125,v1:172.21.15.46:6803/2762665125]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-26T21:27:08.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:08 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:27:08.804 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 26 21:27:08 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-0[43892]: 2024-04-26T21:27:08.675+0000 7fa009c88700 -1 osd.0 0 waiting for initial osdmap 2024-04-26T21:27:08.804 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 26 21:27:08 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-0[43892]: 2024-04-26T21:27:08.678+0000 7fa005a7b700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T21:27:09.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:09 smithi046 ceph-mon[29201]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:27:09.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:09 smithi046 ceph-mon[29201]: from='osd.0 [v2:172.21.15.46:6802/2762665125,v1:172.21.15.46:6803/2762665125]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-26T21:27:09.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:09 smithi046 ceph-mon[29201]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T21:27:09.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:09 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:27:09.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:09 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:27:09.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:09 smithi050 ceph-mon[32753]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:27:09.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:09 smithi050 ceph-mon[32753]: from='osd.0 [v2:172.21.15.46:6802/2762665125,v1:172.21.15.46:6803/2762665125]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-26T21:27:09.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:09 smithi050 ceph-mon[32753]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T21:27:09.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:09 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:27:09.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:09 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:27:10.026 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T21:27:10.026 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T21:27:10.026 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10+0 records in 2024-04-26T21:27:10.026 INFO:teuthology.orchestra.run.smithi050.stderr:10+0 records out 2024-04-26T21:27:10.026 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00832333 s, 1.3 GB/s 2024-04-26T21:27:10.027 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping successful for: 2024-04-26T21:27:10.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:09 smithi142 ceph-mon[32944]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:27:10.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:09 smithi142 ceph-mon[32944]: from='osd.0 [v2:172.21.15.46:6802/2762665125,v1:172.21.15.46:6803/2762665125]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-26T21:27:10.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:09 smithi142 ceph-mon[32944]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T21:27:10.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:09 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:27:10.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:09 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:27:10.640 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi050:vg_nvme/lv_4 2024-04-26T21:27:10.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:10 smithi050 ceph-mon[32753]: purged_snaps scrub starts 2024-04-26T21:27:10.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:10 smithi050 ceph-mon[32753]: purged_snaps scrub ok 2024-04-26T21:27:10.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:10 smithi050 ceph-mon[32753]: osd.0 [v2:172.21.15.46:6802/2762665125,v1:172.21.15.46:6803/2762665125] boot 2024-04-26T21:27:10.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:10 smithi050 ceph-mon[32753]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T21:27:10.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:10 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:27:10.885 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T21:27:11.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:10 smithi142 ceph-mon[32944]: purged_snaps scrub starts 2024-04-26T21:27:11.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:10 smithi142 ceph-mon[32944]: purged_snaps scrub ok 2024-04-26T21:27:11.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:10 smithi142 ceph-mon[32944]: osd.0 [v2:172.21.15.46:6802/2762665125,v1:172.21.15.46:6803/2762665125] boot 2024-04-26T21:27:11.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:10 smithi142 ceph-mon[32944]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T21:27:11.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:10 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:27:11.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:10 smithi046 ceph-mon[29201]: purged_snaps scrub starts 2024-04-26T21:27:11.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:10 smithi046 ceph-mon[29201]: purged_snaps scrub ok 2024-04-26T21:27:11.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:10 smithi046 ceph-mon[29201]: osd.0 [v2:172.21.15.46:6802/2762665125,v1:172.21.15.46:6803/2762665125] boot 2024-04-26T21:27:11.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:10 smithi046 ceph-mon[29201]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T21:27:11.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:10 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:27:11.944 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:11 smithi050 ceph-mon[32753]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:27:11.945 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:11 smithi050 ceph-mon[32753]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T21:27:11.945 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:11 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:11.945 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:11 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:11.945 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:11 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:27:11.945 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:11 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:11.945 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:11 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:11.945 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:11 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:11.945 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:11 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:12.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:11 smithi142 ceph-mon[32944]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:27:12.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:11 smithi142 ceph-mon[32944]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T21:27:12.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:11 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:12.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:11 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:12.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:11 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:27:12.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:11 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:12.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:11 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:12.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:11 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:12.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:11 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:12.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:11 smithi046 ceph-mon[29201]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:27:12.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:11 smithi046 ceph-mon[29201]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T21:27:12.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:11 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:12.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:11 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:12.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:11 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:27:12.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:11 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:12.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:11 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:12.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:11 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:12.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:11 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:12.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.445+0000 7fd1546fc700 1 -- 172.21.15.50:0/1539599739 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd14c102d50 msgr2=0x7fd14c1031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:12.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.445+0000 7fd1546fc700 1 --2- 172.21.15.50:0/1539599739 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd14c102d50 0x7fd14c1031b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd13c0099c0 tx=0x7fd13c009cd0 comp rx=0 tx=0).stop 2024-04-26T21:27:12.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.445+0000 7fd1546fc700 1 -- 172.21.15.50:0/1539599739 shutdown_connections 2024-04-26T21:27:12.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.445+0000 7fd1546fc700 1 --2- 172.21.15.50:0/1539599739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd14c1036f0 0x7fd14c109fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:12.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.445+0000 7fd1546fc700 1 --2- 172.21.15.50:0/1539599739 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd14c101b50 0x7fd14c101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:12.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.445+0000 7fd1546fc700 1 --2- 172.21.15.50:0/1539599739 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd14c102d50 0x7fd14c1031b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:12.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.445+0000 7fd1546fc700 1 -- 172.21.15.50:0/1539599739 >> 172.21.15.50:0/1539599739 conn(0x7fd14c0fd0f0 msgr2=0x7fd14c0ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:12.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.446+0000 7fd1546fc700 1 -- 172.21.15.50:0/1539599739 shutdown_connections 2024-04-26T21:27:12.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.446+0000 7fd1546fc700 1 -- 172.21.15.50:0/1539599739 wait complete. 2024-04-26T21:27:12.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.446+0000 7fd1546fc700 1 Processor -- start 2024-04-26T21:27:12.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.446+0000 7fd1546fc700 1 -- start start 2024-04-26T21:27:12.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.446+0000 7fd1546fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd14c101b50 0x7fd14c0ff2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:12.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.446+0000 7fd1546fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd14c102d50 0x7fd14c19e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:12.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.446+0000 7fd1546fc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd14c1036f0 0x7fd14c1a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:12.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.446+0000 7fd1546fc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd14c10d5c0 con 0x7fd14c102d50 2024-04-26T21:27:12.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.446+0000 7fd1546fc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd14c10d440 con 0x7fd14c101b50 2024-04-26T21:27:12.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.446+0000 7fd1546fc700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd14c10d740 con 0x7fd14c1036f0 2024-04-26T21:27:12.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.446+0000 7fd152498700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd14c101b50 0x7fd14c0ff2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:12.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.447+0000 7fd152498700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd14c101b50 0x7fd14c0ff2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49498/0 (socket says 172.21.15.50:49498) 2024-04-26T21:27:12.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.447+0000 7fd152498700 1 -- 172.21.15.50:0/1451738058 learned_addr learned my addr 172.21.15.50:0/1451738058 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T21:27:12.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.447+0000 7fd152c99700 1 --2- 172.21.15.50:0/1451738058 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd14c1036f0 0x7fd14c1a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:12.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.447+0000 7fd151c97700 1 --2- 172.21.15.50:0/1451738058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd14c102d50 0x7fd14c19e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:12.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.447+0000 7fd152498700 1 -- 172.21.15.50:0/1451738058 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd14c1036f0 msgr2=0x7fd14c1a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:12.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.447+0000 7fd152498700 1 --2- 172.21.15.50:0/1451738058 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd14c1036f0 0x7fd14c1a0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:12.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.447+0000 7fd152498700 1 -- 172.21.15.50:0/1451738058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd14c102d50 msgr2=0x7fd14c19e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:12.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.447+0000 7fd152498700 1 --2- 172.21.15.50:0/1451738058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd14c102d50 0x7fd14c19e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:12.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.447+0000 7fd152498700 1 -- 172.21.15.50:0/1451738058 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd14c1a11f0 con 0x7fd14c101b50 2024-04-26T21:27:12.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.447+0000 7fd151c97700 1 --2- 172.21.15.50:0/1451738058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd14c102d50 0x7fd14c19e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:27:12.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.447+0000 7fd152498700 1 --2- 172.21.15.50:0/1451738058 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd14c101b50 0x7fd14c0ff2e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd14800d980 tx=0x7fd14800dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:12.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.447+0000 7fd1437fe700 1 -- 172.21.15.50:0/1451738058 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd148017030 con 0x7fd14c101b50 2024-04-26T21:27:12.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.447+0000 7fd1437fe700 1 -- 172.21.15.50:0/1451738058 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1480044d0 con 0x7fd14c101b50 2024-04-26T21:27:12.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.447+0000 7fd1546fc700 1 -- 172.21.15.50:0/1451738058 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd14c1a14e0 con 0x7fd14c101b50 2024-04-26T21:27:12.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.447+0000 7fd1437fe700 1 -- 172.21.15.50:0/1451738058 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1480039e0 con 0x7fd14c101b50 2024-04-26T21:27:12.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.447+0000 7fd1546fc700 1 -- 172.21.15.50:0/1451738058 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd14c1a1a70 con 0x7fd14c101b50 2024-04-26T21:27:12.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.448+0000 7fd1546fc700 1 -- 172.21.15.50:0/1451738058 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd14c06a110 con 0x7fd14c101b50 2024-04-26T21:27:12.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.449+0000 7fd1437fe700 1 -- 172.21.15.50:0/1451738058 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fd14801a070 con 0x7fd14c101b50 2024-04-26T21:27:12.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.450+0000 7fd1437fe700 1 --2- 172.21.15.50:0/1451738058 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fd1380a3c30 0x7fd1380a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:12.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.450+0000 7fd1437fe700 1 -- 172.21.15.50:0/1451738058 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fd1480ce670 con 0x7fd14c101b50 2024-04-26T21:27:12.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.450+0000 7fd151c97700 1 --2- 172.21.15.50:0/1451738058 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fd1380a3c30 0x7fd1380a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:12.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.451+0000 7fd151c97700 1 --2- 172.21.15.50:0/1451738058 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fd1380a3c30 0x7fd1380a60f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009fd0 tx=0x7fd13c005e80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:12.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.454+0000 7fd1437fe700 1 -- 172.21.15.50:0/1451738058 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd14800e350 con 0x7fd14c101b50 2024-04-26T21:27:12.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:12.668+0000 7fd1546fc700 1 -- 172.21.15.50:0/1451738058 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fd14c063860 con 0x7fd1380a3c30 2024-04-26T21:27:13.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:12 smithi142 ceph-mon[32944]: Detected new or changed devices on smithi046 2024-04-26T21:27:13.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:12 smithi142 ceph-mon[32944]: Adjusting osd_memory_target on smithi046 to 17093M 2024-04-26T21:27:13.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:12 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:27:13.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:12 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:27:13.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:12 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:13.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:12 smithi046 ceph-mon[29201]: Detected new or changed devices on smithi046 2024-04-26T21:27:13.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:12 smithi046 ceph-mon[29201]: Adjusting osd_memory_target on smithi046 to 17093M 2024-04-26T21:27:13.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:12 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:27:13.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:12 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:27:13.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:12 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:13.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:12 smithi050 ceph-mon[32753]: Detected new or changed devices on smithi046 2024-04-26T21:27:13.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:12 smithi050 ceph-mon[32753]: Adjusting osd_memory_target on smithi046 to 17093M 2024-04-26T21:27:13.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:12 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:27:13.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:12 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:27:13.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:12 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:13.961 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:13 smithi050 ceph-mon[32753]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:13.962 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:13 smithi050 ceph-mon[32753]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:27:14.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:13 smithi142 ceph-mon[32944]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:14.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:13 smithi142 ceph-mon[32944]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:27:14.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:13 smithi046 ceph-mon[29201]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:14.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:13 smithi046 ceph-mon[29201]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:27:16.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:15 smithi142 ceph-mon[32944]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:16.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:15 smithi142 ceph-mon[32944]: from='client.? 172.21.15.50:0/1733362612' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc4d6f52-fc8a-4efd-80c9-c092b57e7f42"}]: dispatch 2024-04-26T21:27:16.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:15 smithi142 ceph-mon[32944]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc4d6f52-fc8a-4efd-80c9-c092b57e7f42"}]: dispatch 2024-04-26T21:27:16.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:15 smithi142 ceph-mon[32944]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc4d6f52-fc8a-4efd-80c9-c092b57e7f42"}]': finished 2024-04-26T21:27:16.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:15 smithi142 ceph-mon[32944]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T21:27:16.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:15 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:27:16.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:15 smithi046 ceph-mon[29201]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:16.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:15 smithi046 ceph-mon[29201]: from='client.? 172.21.15.50:0/1733362612' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc4d6f52-fc8a-4efd-80c9-c092b57e7f42"}]: dispatch 2024-04-26T21:27:16.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:15 smithi046 ceph-mon[29201]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc4d6f52-fc8a-4efd-80c9-c092b57e7f42"}]: dispatch 2024-04-26T21:27:16.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:15 smithi046 ceph-mon[29201]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc4d6f52-fc8a-4efd-80c9-c092b57e7f42"}]': finished 2024-04-26T21:27:16.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:15 smithi046 ceph-mon[29201]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T21:27:16.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:15 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:27:16.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:15 smithi050 ceph-mon[32753]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:16.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:15 smithi050 ceph-mon[32753]: from='client.? 172.21.15.50:0/1733362612' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc4d6f52-fc8a-4efd-80c9-c092b57e7f42"}]: dispatch 2024-04-26T21:27:16.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:15 smithi050 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc4d6f52-fc8a-4efd-80c9-c092b57e7f42"}]: dispatch 2024-04-26T21:27:16.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:15 smithi050 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc4d6f52-fc8a-4efd-80c9-c092b57e7f42"}]': finished 2024-04-26T21:27:16.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:15 smithi050 ceph-mon[32753]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T21:27:16.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:15 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:27:16.979 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:16 smithi046 ceph-mon[29201]: from='client.? 172.21.15.50:0/1682029438' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:27:17.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:16 smithi142 ceph-mon[32944]: from='client.? 172.21.15.50:0/1682029438' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:27:17.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:16 smithi050 ceph-mon[32753]: from='client.? 172.21.15.50:0/1682029438' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:27:18.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:17 smithi142 ceph-mon[32944]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:18.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:17 smithi046 ceph-mon[29201]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:18.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:17 smithi050 ceph-mon[32753]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:19.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:19 smithi050 ceph-mon[32753]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:20.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:19 smithi142 ceph-mon[32944]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:20.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:19 smithi046 ceph-mon[29201]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:22.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:21 smithi142 ceph-mon[32944]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:22.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:21 smithi046 ceph-mon[29201]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:22.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:21 smithi050 ceph-mon[32753]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:23.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:22 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T21:27:23.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:22 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:23.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:22 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T21:27:23.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:22 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:23.083 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:22 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T21:27:23.083 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:22 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:23.971 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:23 smithi046 ceph-mon[29201]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:23.971 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:23 smithi046 ceph-mon[29201]: Deploying daemon osd.1 on smithi050 2024-04-26T21:27:23.997 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:23 smithi050 ceph-mon[32753]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:23.997 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:23 smithi050 ceph-mon[32753]: Deploying daemon osd.1 on smithi050 2024-04-26T21:27:24.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:23 smithi142 ceph-mon[32944]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:24.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:23 smithi142 ceph-mon[32944]: Deploying daemon osd.1 on smithi050 2024-04-26T21:27:26.028 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:25 smithi050 ceph-mon[32753]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:26.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:25 smithi142 ceph-mon[32944]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:26.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:25 smithi046 ceph-mon[29201]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:27.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:27 smithi050 ceph-mon[32753]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:28.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:27 smithi142 ceph-mon[32944]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:28.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:27 smithi046 ceph-mon[29201]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:28.986 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:28 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:28.987 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:28 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:28.987 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:28 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:27:29.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:28 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:29.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:28 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:29.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:28 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:27:29.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:28 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:29.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:28 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:29.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:28 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:27:30.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:29 smithi142 ceph-mon[32944]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:30.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:29 smithi046 ceph-mon[29201]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:30.101 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:29 smithi050 ceph-mon[32753]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:30.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:30.784+0000 7fd1437fe700 1 -- 172.21.15.50:0/1451738058 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd14c063860 con 0x7fd1380a3c30 2024-04-26T21:27:30.785 INFO:teuthology.orchestra.run.smithi050.stdout:Created osd(s) 1 on host 'smithi050' 2024-04-26T21:27:30.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:30.786+0000 7fd1546fc700 1 -- 172.21.15.50:0/1451738058 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fd1380a3c30 msgr2=0x7fd1380a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:30.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:30.786+0000 7fd1546fc700 1 --2- 172.21.15.50:0/1451738058 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fd1380a3c30 0x7fd1380a60f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009fd0 tx=0x7fd13c005e80 comp rx=0 tx=0).stop 2024-04-26T21:27:30.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:30.786+0000 7fd1546fc700 1 -- 172.21.15.50:0/1451738058 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd14c101b50 msgr2=0x7fd14c0ff2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:30.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:30.786+0000 7fd1546fc700 1 --2- 172.21.15.50:0/1451738058 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd14c101b50 0x7fd14c0ff2e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd14800d980 tx=0x7fd14800dc90 comp rx=0 tx=0).stop 2024-04-26T21:27:30.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:30.786+0000 7fd1546fc700 1 -- 172.21.15.50:0/1451738058 shutdown_connections 2024-04-26T21:27:30.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:30.786+0000 7fd1546fc700 1 --2- 172.21.15.50:0/1451738058 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd14c1036f0 0x7fd14c1a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:30.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:30.786+0000 7fd1546fc700 1 --2- 172.21.15.50:0/1451738058 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd14c101b50 0x7fd14c0ff2e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:30.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:30.786+0000 7fd1546fc700 1 --2- 172.21.15.50:0/1451738058 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fd1380a3c30 0x7fd1380a60f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:30.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:30.786+0000 7fd1546fc700 1 --2- 172.21.15.50:0/1451738058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd14c102d50 0x7fd14c19e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:30.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:30.786+0000 7fd1546fc700 1 -- 172.21.15.50:0/1451738058 >> 172.21.15.50:0/1451738058 conn(0x7fd14c0fd0f0 msgr2=0x7fd14c106480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:30.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:30.786+0000 7fd1546fc700 1 -- 172.21.15.50:0/1451738058 shutdown_connections 2024-04-26T21:27:30.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:27:30.786+0000 7fd1546fc700 1 -- 172.21.15.50:0/1451738058 wait complete. 2024-04-26T21:27:31.762 DEBUG:teuthology.orchestra.run.smithi050:osd.1> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.1.service 2024-04-26T21:27:31.764 INFO:tasks.cephadm:Deploying osd.2 on smithi142 with /dev/vg_nvme/lv_4... 2024-04-26T21:27:31.765 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T21:27:32.009 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:27:32.034 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:31 smithi050 ceph-mon[32753]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:32.034 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:32.035 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:32.035 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:32.035 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:32.035 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:31 smithi050 ceph-mon[32753]: from='osd.1 [v2:172.21.15.50:6800/1628008970,v1:172.21.15.50:6801/1628008970]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T21:27:32.035 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:31 smithi050 ceph-mon[32753]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T21:27:32.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:31 smithi142 ceph-mon[32944]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:32.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:32.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:32.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:32.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:32.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:31 smithi142 ceph-mon[32944]: from='osd.1 [v2:172.21.15.50:6800/1628008970,v1:172.21.15.50:6801/1628008970]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T21:27:32.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:31 smithi142 ceph-mon[32944]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T21:27:32.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:31 smithi046 ceph-mon[29201]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:32.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:32.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:32.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:32.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:32.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:31 smithi046 ceph-mon[29201]: from='osd.1 [v2:172.21.15.50:6800/1628008970,v1:172.21.15.50:6801/1628008970]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T21:27:32.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:31 smithi046 ceph-mon[29201]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T21:27:33.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:33 smithi050 ceph-mon[32753]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:33.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:33 smithi050 ceph-mon[32753]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T21:27:33.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:33 smithi050 ceph-mon[32753]: from='osd.1 [v2:172.21.15.50:6800/1628008970,v1:172.21.15.50:6801/1628008970]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T21:27:33.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:33 smithi050 ceph-mon[32753]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T21:27:33.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:33 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:27:33.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:33 smithi050 ceph-mon[32753]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T21:27:33.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:33 smithi142 ceph-mon[32944]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:33.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:33 smithi142 ceph-mon[32944]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T21:27:33.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:33 smithi142 ceph-mon[32944]: from='osd.1 [v2:172.21.15.50:6800/1628008970,v1:172.21.15.50:6801/1628008970]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T21:27:33.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:33 smithi142 ceph-mon[32944]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T21:27:33.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:33 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:27:33.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:33 smithi142 ceph-mon[32944]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T21:27:33.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:33 smithi046 ceph-mon[29201]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:33.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:33 smithi046 ceph-mon[29201]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T21:27:33.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:33 smithi046 ceph-mon[29201]: from='osd.1 [v2:172.21.15.50:6800/1628008970,v1:172.21.15.50:6801/1628008970]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T21:27:33.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:33 smithi046 ceph-mon[29201]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T21:27:33.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:33 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:27:33.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:33 smithi046 ceph-mon[29201]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T21:27:33.854 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T21:27:33.854 INFO:teuthology.orchestra.run.smithi142.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T21:27:33.854 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10+0 records in 2024-04-26T21:27:33.854 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records out 2024-04-26T21:27:33.854 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113858 s, 921 MB/s 2024-04-26T21:27:33.854 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping successful for: 2024-04-26T21:27:34.369 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 21:27:34 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-1[37343]: 2024-04-26T21:27:34.008+0000 7f4f549b1700 -1 osd.1 0 waiting for initial osdmap 2024-04-26T21:27:34.369 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 21:27:34 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-1[37343]: 2024-04-26T21:27:34.012+0000 7f4f4c79c700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T21:27:34.446 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi142:vg_nvme/lv_4 2024-04-26T21:27:34.684 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:27:34.697 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:34 smithi046 ceph-mon[29201]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T21:27:34.697 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:34 smithi046 ceph-mon[29201]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T21:27:34.697 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:34 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:27:34.698 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:34 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:27:34.698 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:34 smithi046 ceph-mon[29201]: from='osd.1 ' entity='osd.1' 2024-04-26T21:27:34.732 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:34 smithi142 ceph-mon[32944]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T21:27:34.732 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:34 smithi142 ceph-mon[32944]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T21:27:34.732 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:34 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:27:34.732 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:34 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:27:34.732 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:34 smithi142 ceph-mon[32944]: from='osd.1 ' entity='osd.1' 2024-04-26T21:27:34.848 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:34 smithi050 ceph-mon[32753]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T21:27:34.848 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:34 smithi050 ceph-mon[32753]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T21:27:34.848 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:34 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:27:34.848 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:34 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:27:34.848 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:34 smithi050 ceph-mon[32753]: from='osd.1 ' entity='osd.1' 2024-04-26T21:27:35.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:35 smithi046 ceph-mon[29201]: purged_snaps scrub starts 2024-04-26T21:27:35.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:35 smithi046 ceph-mon[29201]: purged_snaps scrub ok 2024-04-26T21:27:35.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:35 smithi046 ceph-mon[29201]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:35.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:27:35.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:35.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:35.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:27:35.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:35.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:35.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:35.555 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:35.555 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:35 smithi046 ceph-mon[29201]: osd.1 [v2:172.21.15.50:6800/1628008970,v1:172.21.15.50:6801/1628008970] boot 2024-04-26T21:27:35.555 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:35 smithi046 ceph-mon[29201]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T21:27:35.555 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:27:35.555 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:27:35.555 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:35.555 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:35.555 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:35 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:35.680 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:35 smithi142 ceph-mon[32944]: purged_snaps scrub starts 2024-04-26T21:27:35.680 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:35 smithi142 ceph-mon[32944]: purged_snaps scrub ok 2024-04-26T21:27:35.680 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:35 smithi142 ceph-mon[32944]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:35.680 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:27:35.680 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:35.680 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:35.680 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:27:35.680 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:35.680 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:35.681 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:35.681 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:35.681 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:35 smithi142 ceph-mon[32944]: osd.1 [v2:172.21.15.50:6800/1628008970,v1:172.21.15.50:6801/1628008970] boot 2024-04-26T21:27:35.681 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:35 smithi142 ceph-mon[32944]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T21:27:35.681 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:27:35.681 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:27:35.681 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:35.681 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:35.681 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:35 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:35.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:35 smithi050 ceph-mon[32753]: purged_snaps scrub starts 2024-04-26T21:27:35.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:35 smithi050 ceph-mon[32753]: purged_snaps scrub ok 2024-04-26T21:27:35.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:35 smithi050 ceph-mon[32753]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:27:35.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:27:35.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:35.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:35.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:27:35.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:35.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:35.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:35.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:35.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:35 smithi050 ceph-mon[32753]: osd.1 [v2:172.21.15.50:6800/1628008970,v1:172.21.15.50:6801/1628008970] boot 2024-04-26T21:27:35.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:35 smithi050 ceph-mon[32753]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T21:27:35.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:27:35.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:27:35.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:35.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:35.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:35 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:36.299 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.296+0000 7f860eb62700 1 -- 172.21.15.142:0/2560358438 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8608103f40 msgr2=0x7f860810a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:36.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.296+0000 7f860eb62700 1 --2- 172.21.15.142:0/2560358438 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8608103f40 0x7f860810a7f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f860000b0a0 tx=0x7f860000b3b0 comp rx=0 tx=0).stop 2024-04-26T21:27:36.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.296+0000 7f860eb62700 1 -- 172.21.15.142:0/2560358438 shutdown_connections 2024-04-26T21:27:36.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.296+0000 7f860eb62700 1 --2- 172.21.15.142:0/2560358438 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8608103f40 0x7f860810a7f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:36.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.296+0000 7f860eb62700 1 --2- 172.21.15.142:0/2560358438 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86081023a0 0x7f86081027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:36.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.296+0000 7f860eb62700 1 --2- 172.21.15.142:0/2560358438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f86081035a0 0x7f8608103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:36.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.296+0000 7f860eb62700 1 -- 172.21.15.142:0/2560358438 >> 172.21.15.142:0/2560358438 conn(0x7f86080fd980 msgr2=0x7f86080ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:36.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.296+0000 7f860eb62700 1 -- 172.21.15.142:0/2560358438 shutdown_connections 2024-04-26T21:27:36.300 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.296+0000 7f860eb62700 1 -- 172.21.15.142:0/2560358438 wait complete. 2024-04-26T21:27:36.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.297+0000 7f860eb62700 1 Processor -- start 2024-04-26T21:27:36.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.297+0000 7f860eb62700 1 -- start start 2024-04-26T21:27:36.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.297+0000 7f860eb62700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86081023a0 0x7f8608198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:36.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.297+0000 7f860eb62700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86081035a0 0x7f8608198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:36.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.297+0000 7f860eb62700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8608103f40 0x7f860819fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:36.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.297+0000 7f860eb62700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f860810de30 con 0x7f8608103f40 2024-04-26T21:27:36.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.297+0000 7f860eb62700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f860810dcb0 con 0x7f86081023a0 2024-04-26T21:27:36.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.297+0000 7f860eb62700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f860810dfb0 con 0x7f86081035a0 2024-04-26T21:27:36.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.297+0000 7f85fffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86081035a0 0x7f8608198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:36.301 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.297+0000 7f85fffff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86081035a0 0x7f8608198980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:48692/0 (socket says 172.21.15.142:48692) 2024-04-26T21:27:36.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.297+0000 7f85fffff700 1 -- 172.21.15.142:0/2508483083 learned_addr learned my addr 172.21.15.142:0/2508483083 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-04-26T21:27:36.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.297+0000 7f860d0ff700 1 --2- 172.21.15.142:0/2508483083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8608103f40 0x7f860819fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:36.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.297+0000 7f860c8fe700 1 --2- 172.21.15.142:0/2508483083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86081023a0 0x7f8608198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:36.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.297+0000 7f85fffff700 1 -- 172.21.15.142:0/2508483083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86081023a0 msgr2=0x7f8608198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:36.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.297+0000 7f85fffff700 1 --2- 172.21.15.142:0/2508483083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86081023a0 0x7f8608198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:36.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.297+0000 7f85fffff700 1 -- 172.21.15.142:0/2508483083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8608103f40 msgr2=0x7f860819fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:36.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.297+0000 7f85fffff700 1 --2- 172.21.15.142:0/2508483083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8608103f40 0x7f860819fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:36.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.297+0000 7f85fffff700 1 -- 172.21.15.142:0/2508483083 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86081a00a0 con 0x7f86081035a0 2024-04-26T21:27:36.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.297+0000 7f85fffff700 1 --2- 172.21.15.142:0/2508483083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86081035a0 0x7f8608198980 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f85f80099a0 tx=0x7f85f8009cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:36.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.298+0000 7f85fdffb700 1 -- 172.21.15.142:0/2508483083 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f801b070 con 0x7f86081035a0 2024-04-26T21:27:36.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.298+0000 7f85fdffb700 1 -- 172.21.15.142:0/2508483083 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85f800d260 con 0x7f86081035a0 2024-04-26T21:27:36.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.298+0000 7f860eb62700 1 -- 172.21.15.142:0/2508483083 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f860806bbf0 con 0x7f86081035a0 2024-04-26T21:27:36.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.298+0000 7f85fdffb700 1 -- 172.21.15.142:0/2508483083 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f800c9c0 con 0x7f86081035a0 2024-04-26T21:27:36.302 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.298+0000 7f860eb62700 1 -- 172.21.15.142:0/2508483083 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f860806c180 con 0x7f86081035a0 2024-04-26T21:27:36.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.299+0000 7f860eb62700 1 -- 172.21.15.142:0/2508483083 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f860806a110 con 0x7f86081035a0 2024-04-26T21:27:36.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.300+0000 7f85fdffb700 1 -- 172.21.15.142:0/2508483083 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f85f801e070 con 0x7f86081035a0 2024-04-26T21:27:36.303 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.300+0000 7f85fdffb700 1 --2- 172.21.15.142:0/2508483083 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f85f00a3c30 0x7f85f00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:36.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.301+0000 7f85fdffb700 1 -- 172.21.15.142:0/2508483083 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2309+0+0 (secure 0 0 0) 0x7f85f80cf820 con 0x7f86081035a0 2024-04-26T21:27:36.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.301+0000 7f860c8fe700 1 --2- 172.21.15.142:0/2508483083 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f85f00a3c30 0x7f85f00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:36.304 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.301+0000 7f860c8fe700 1 --2- 172.21.15.142:0/2508483083 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f85f00a3c30 0x7f85f00a60f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8608103400 tx=0x7f860400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:36.306 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.305+0000 7f85fdffb700 1 -- 172.21.15.142:0/2508483083 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f85f809da50 con 0x7f86081035a0 2024-04-26T21:27:36.515 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:36.514+0000 7f860eb62700 1 -- 172.21.15.142:0/2508483083 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f860806c3e0 con 0x7f85f00a3c30 2024-04-26T21:27:36.515 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:36 smithi142 ceph-mon[32944]: Detected new or changed devices on smithi050 2024-04-26T21:27:36.515 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:36 smithi142 ceph-mon[32944]: Adjusting osd_memory_target on smithi050 to 17093M 2024-04-26T21:27:36.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:36 smithi046 ceph-mon[29201]: Detected new or changed devices on smithi050 2024-04-26T21:27:36.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:36 smithi046 ceph-mon[29201]: Adjusting osd_memory_target on smithi050 to 17093M 2024-04-26T21:27:36.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:36 smithi050 ceph-mon[32753]: Detected new or changed devices on smithi050 2024-04-26T21:27:36.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:36 smithi050 ceph-mon[32753]: Adjusting osd_memory_target on smithi050 to 17093M 2024-04-26T21:27:37.633 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:37 smithi142 ceph-mon[32944]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:37.633 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:37 smithi142 ceph-mon[32944]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T21:27:37.633 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:37 smithi142 ceph-mon[32944]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:27:37.633 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:37 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:27:37.633 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:37 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:27:37.634 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:37 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:37.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:37 smithi046 ceph-mon[29201]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:37.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:37 smithi046 ceph-mon[29201]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T21:27:37.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:37 smithi046 ceph-mon[29201]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:27:37.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:37 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:27:37.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:37 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:27:37.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:37 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:37.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:37 smithi050 ceph-mon[32753]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:37.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:37 smithi050 ceph-mon[32753]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T21:27:37.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:37 smithi050 ceph-mon[32753]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:27:37.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:37 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:27:37.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:37 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:27:37.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:37 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:39.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:39 smithi142 ceph-mon[32944]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:39.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:39 smithi142 ceph-mon[32944]: from='client.? 172.21.15.142:0/1593743798' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26d1baa1-4ea1-4b7e-9fcf-4f8bbf7329c5"}]: dispatch 2024-04-26T21:27:39.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:39 smithi142 ceph-mon[32944]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26d1baa1-4ea1-4b7e-9fcf-4f8bbf7329c5"}]: dispatch 2024-04-26T21:27:39.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:39 smithi142 ceph-mon[32944]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26d1baa1-4ea1-4b7e-9fcf-4f8bbf7329c5"}]': finished 2024-04-26T21:27:39.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:39 smithi142 ceph-mon[32944]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T21:27:39.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:39 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:27:39.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:39 smithi046 ceph-mon[29201]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:39.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:39 smithi046 ceph-mon[29201]: from='client.? 172.21.15.142:0/1593743798' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26d1baa1-4ea1-4b7e-9fcf-4f8bbf7329c5"}]: dispatch 2024-04-26T21:27:39.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:39 smithi046 ceph-mon[29201]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26d1baa1-4ea1-4b7e-9fcf-4f8bbf7329c5"}]: dispatch 2024-04-26T21:27:39.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:39 smithi046 ceph-mon[29201]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26d1baa1-4ea1-4b7e-9fcf-4f8bbf7329c5"}]': finished 2024-04-26T21:27:39.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:39 smithi046 ceph-mon[29201]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T21:27:39.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:39 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:27:39.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:39 smithi050 ceph-mon[32753]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:39.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:39 smithi050 ceph-mon[32753]: from='client.? 172.21.15.142:0/1593743798' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26d1baa1-4ea1-4b7e-9fcf-4f8bbf7329c5"}]: dispatch 2024-04-26T21:27:39.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:39 smithi050 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26d1baa1-4ea1-4b7e-9fcf-4f8bbf7329c5"}]: dispatch 2024-04-26T21:27:39.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:39 smithi050 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26d1baa1-4ea1-4b7e-9fcf-4f8bbf7329c5"}]': finished 2024-04-26T21:27:39.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:39 smithi050 ceph-mon[32753]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T21:27:39.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:39 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:27:40.692 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:40 smithi142 ceph-mon[32944]: from='client.? 172.21.15.142:0/2229310591' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:27:40.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:40 smithi046 ceph-mon[29201]: from='client.? 172.21.15.142:0/2229310591' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:27:40.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:40 smithi050 ceph-mon[32753]: from='client.? 172.21.15.142:0/2229310591' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:27:41.703 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:41 smithi142 ceph-mon[32944]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:41.769 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:41 smithi050 ceph-mon[32753]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:41.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:41 smithi046 ceph-mon[29201]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:43.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:43 smithi142 ceph-mon[32944]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:43.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:43 smithi046 ceph-mon[29201]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:43.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:43 smithi050 ceph-mon[32753]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:45.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:45 smithi142 ceph-mon[32944]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:45.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:45 smithi046 ceph-mon[29201]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:45.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:45 smithi050 ceph-mon[32753]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:46.713 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:46 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T21:27:46.714 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:46 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:46.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:46 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T21:27:46.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:46 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:46.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:46 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T21:27:46.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:46 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:47.642 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:47 smithi142 ceph-mon[32944]: Deploying daemon osd.2 on smithi142 2024-04-26T21:27:47.642 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:47 smithi142 ceph-mon[32944]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:47.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:47 smithi046 ceph-mon[29201]: Deploying daemon osd.2 on smithi142 2024-04-26T21:27:47.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:47 smithi046 ceph-mon[29201]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:47.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:47 smithi050 ceph-mon[32753]: Deploying daemon osd.2 on smithi142 2024-04-26T21:27:47.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:47 smithi050 ceph-mon[32753]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:49.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:49 smithi142 ceph-mon[32944]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:49.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:49 smithi046 ceph-mon[29201]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:49.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:49 smithi050 ceph-mon[32753]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:51.700 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:51 smithi142 ceph-mon[32944]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:51.700 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:51 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:51.701 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:51 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:51.701 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:51 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:27:51.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:51 smithi046 ceph-mon[29201]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:51.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:51 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:51.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:51 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:51.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:51 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:27:51.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:51 smithi050 ceph-mon[32753]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:51.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:51 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:51.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:51 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:51.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:51 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:27:53.619 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:53 smithi142 ceph-mon[32944]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:53.620 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:53 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:53.620 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:53 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:53.620 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:53 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:53.620 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:53 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:53.620 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:53 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:53.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:53 smithi046 ceph-mon[29201]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:53.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:53 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:53.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:53 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:53.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:53 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:53.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:53 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:53.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:53 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:53.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:53 smithi050 ceph-mon[32753]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:53.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:53 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:53.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:53 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:53.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:53 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:53.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:53 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:53.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:53 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:54.528 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:54.527+0000 7f85fdffb700 1 -- 172.21.15.142:0/2508483083 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f860806c3e0 con 0x7f85f00a3c30 2024-04-26T21:27:54.528 INFO:teuthology.orchestra.run.smithi142.stdout:Created osd(s) 2 on host 'smithi142' 2024-04-26T21:27:54.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:54.528+0000 7f860eb62700 1 -- 172.21.15.142:0/2508483083 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f85f00a3c30 msgr2=0x7f85f00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:54.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:54.528+0000 7f860eb62700 1 --2- 172.21.15.142:0/2508483083 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f85f00a3c30 0x7f85f00a60f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8608103400 tx=0x7f860400b410 comp rx=0 tx=0).stop 2024-04-26T21:27:54.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:54.528+0000 7f860eb62700 1 -- 172.21.15.142:0/2508483083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86081035a0 msgr2=0x7f8608198980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:54.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:54.528+0000 7f860eb62700 1 --2- 172.21.15.142:0/2508483083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86081035a0 0x7f8608198980 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f85f80099a0 tx=0x7f85f8009cb0 comp rx=0 tx=0).stop 2024-04-26T21:27:54.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:54.528+0000 7f860eb62700 1 -- 172.21.15.142:0/2508483083 shutdown_connections 2024-04-26T21:27:54.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:54.528+0000 7f860eb62700 1 --2- 172.21.15.142:0/2508483083 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f85f00a3c30 0x7f85f00a60f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:54.530 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:54.528+0000 7f860eb62700 1 --2- 172.21.15.142:0/2508483083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8608103f40 0x7f860819fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:54.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:54.528+0000 7f860eb62700 1 --2- 172.21.15.142:0/2508483083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f86081023a0 0x7f8608198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:54.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:54.528+0000 7f860eb62700 1 --2- 172.21.15.142:0/2508483083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f86081035a0 0x7f8608198980 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:54.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:54.528+0000 7f860eb62700 1 -- 172.21.15.142:0/2508483083 >> 172.21.15.142:0/2508483083 conn(0x7f86080fd980 msgr2=0x7f8608106920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:54.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:54.528+0000 7f860eb62700 1 -- 172.21.15.142:0/2508483083 shutdown_connections 2024-04-26T21:27:54.531 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:27:54.528+0000 7f860eb62700 1 -- 172.21.15.142:0/2508483083 wait complete. 2024-04-26T21:27:55.183 DEBUG:teuthology.orchestra.run.smithi142:osd.2> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.2.service 2024-04-26T21:27:55.185 INFO:tasks.cephadm:Deploying osd.3 on smithi046 with /dev/vg_nvme/lv_3... 2024-04-26T21:27:55.185 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T21:27:55.433 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:27:55.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:55 smithi142 ceph-mon[32944]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:55.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:55 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:55.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:55 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:55.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:55 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:27:55.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:55 smithi142 ceph-mon[32944]: from='osd.2 [v2:172.21.15.142:6800/1513279137,v1:172.21.15.142:6801/1513279137]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T21:27:55.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:55 smithi142 ceph-mon[32944]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T21:27:55.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:55 smithi046 ceph-mon[29201]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:55.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:55 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:55.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:55 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:55.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:55 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:27:55.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:55 smithi046 ceph-mon[29201]: from='osd.2 [v2:172.21.15.142:6800/1513279137,v1:172.21.15.142:6801/1513279137]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T21:27:55.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:55 smithi046 ceph-mon[29201]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T21:27:55.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:55 smithi050 ceph-mon[32753]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:55.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:55 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:55.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:55 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:55.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:55 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:27:55.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:55 smithi050 ceph-mon[32753]: from='osd.2 [v2:172.21.15.142:6800/1513279137,v1:172.21.15.142:6801/1513279137]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T21:27:55.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:55 smithi050 ceph-mon[32753]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T21:27:56.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:56 smithi142 ceph-mon[32944]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T21:27:56.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:56 smithi142 ceph-mon[32944]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T21:27:56.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:56 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:27:56.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:56 smithi142 ceph-mon[32944]: from='osd.2 [v2:172.21.15.142:6800/1513279137,v1:172.21.15.142:6801/1513279137]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-04-26T21:27:56.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:56 smithi142 ceph-mon[32944]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-04-26T21:27:56.792 INFO:journalctl@ceph.osd.2.smithi142.stdout:Apr 26 21:27:56 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-2[37956]: 2024-04-26T21:27:56.534+0000 7fd8d6165700 -1 osd.2 0 waiting for initial osdmap 2024-04-26T21:27:56.792 INFO:journalctl@ceph.osd.2.smithi142.stdout:Apr 26 21:27:56 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-2[37956]: 2024-04-26T21:27:56.538+0000 7fd8d1f58700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T21:27:56.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:56 smithi046 ceph-mon[29201]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T21:27:56.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:56 smithi046 ceph-mon[29201]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T21:27:56.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:56 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:27:56.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:56 smithi046 ceph-mon[29201]: from='osd.2 [v2:172.21.15.142:6800/1513279137,v1:172.21.15.142:6801/1513279137]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-04-26T21:27:56.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:56 smithi046 ceph-mon[29201]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-04-26T21:27:56.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:56 smithi050 ceph-mon[32753]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T21:27:56.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:56 smithi050 ceph-mon[32753]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T21:27:56.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:56 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:27:56.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:56 smithi050 ceph-mon[32753]: from='osd.2 [v2:172.21.15.142:6800/1513279137,v1:172.21.15.142:6801/1513279137]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-04-26T21:27:56.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:56 smithi050 ceph-mon[32753]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-04-26T21:27:57.643 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T21:27:57.643 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T21:27:57.643 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2024-04-26T21:27:57.643 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2024-04-26T21:27:57.643 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00837055 s, 1.3 GB/s 2024-04-26T21:27:57.643 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2024-04-26T21:27:57.647 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:57 smithi142 ceph-mon[32944]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:57.647 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:57 smithi142 ceph-mon[32944]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-04-26T21:27:57.647 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:57 smithi142 ceph-mon[32944]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T21:27:57.647 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:57 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:27:57.647 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:57 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:27:57.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:57 smithi046 ceph-mon[29201]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:57.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:57 smithi046 ceph-mon[29201]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-04-26T21:27:57.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:57 smithi046 ceph-mon[29201]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T21:27:57.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:57 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:27:57.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:57 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:27:57.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:57 smithi050 ceph-mon[32753]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:27:57.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:57 smithi050 ceph-mon[32753]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-04-26T21:27:57.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:57 smithi050 ceph-mon[32753]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T21:27:57.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:57 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:27:57.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:57 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:27:58.417 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi046:vg_nvme/lv_3 2024-04-26T21:27:58.660 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:27:58.782 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:58 smithi046 ceph-mon[29201]: purged_snaps scrub starts 2024-04-26T21:27:58.782 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:58 smithi046 ceph-mon[29201]: purged_snaps scrub ok 2024-04-26T21:27:58.782 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:58 smithi046 ceph-mon[29201]: osd.2 [v2:172.21.15.142:6800/1513279137,v1:172.21.15.142:6801/1513279137] boot 2024-04-26T21:27:58.782 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:58 smithi046 ceph-mon[29201]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T21:27:58.782 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:58 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:27:58.782 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:58 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T21:27:58.783 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:58 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:58.783 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:58 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:58.783 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:58 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:27:58.783 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:58 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:58.783 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:58 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:58.783 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:58 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:58.783 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:58 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:58.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:58 smithi142 ceph-mon[32944]: purged_snaps scrub starts 2024-04-26T21:27:58.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:58 smithi142 ceph-mon[32944]: purged_snaps scrub ok 2024-04-26T21:27:58.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:58 smithi142 ceph-mon[32944]: osd.2 [v2:172.21.15.142:6800/1513279137,v1:172.21.15.142:6801/1513279137] boot 2024-04-26T21:27:58.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:58 smithi142 ceph-mon[32944]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T21:27:58.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:58 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:27:58.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:58 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T21:27:58.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:58 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:58.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:58 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:58.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:58 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:27:58.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:58 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:58.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:58 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:58.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:58 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:58.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:58 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:58.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:58 smithi050 ceph-mon[32753]: purged_snaps scrub starts 2024-04-26T21:27:58.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:58 smithi050 ceph-mon[32753]: purged_snaps scrub ok 2024-04-26T21:27:58.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:58 smithi050 ceph-mon[32753]: osd.2 [v2:172.21.15.142:6800/1513279137,v1:172.21.15.142:6801/1513279137] boot 2024-04-26T21:27:58.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:58 smithi050 ceph-mon[32753]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T21:27:58.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:58 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:27:58.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:58 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T21:27:58.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:58 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:58.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:58 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:58.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:58 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:27:58.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:58 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:58.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:58 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:58.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:58 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:58.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:58 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:27:59.789 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:59 smithi046 ceph-mon[29201]: pgmap v82: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:27:59.789 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:59 smithi046 ceph-mon[29201]: Detected new or changed devices on smithi142 2024-04-26T21:27:59.789 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:59 smithi046 ceph-mon[29201]: Adjusting osd_memory_target on smithi142 to 17088M 2024-04-26T21:27:59.789 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:59 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T21:27:59.789 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:59 smithi046 ceph-mon[29201]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T21:27:59.789 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:59 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T21:27:59.790 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 26 21:27:59 smithi046 sudo[48279]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T21:27:59.790 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 26 21:27:59 smithi046 sudo[48279]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T21:27:59.790 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 26 21:27:59 smithi046 sudo[48279]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T21:27:59.790 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 26 21:27:59 smithi046 sudo[48279]: pam_unix(sudo:session): session closed for user root 2024-04-26T21:27:59.790 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 26 21:27:59 smithi046 sudo[48288]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T21:27:59.790 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 26 21:27:59 smithi046 sudo[48288]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T21:27:59.790 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 26 21:27:59 smithi046 sudo[48288]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T21:27:59.790 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 26 21:27:59 smithi046 sudo[48288]: pam_unix(sudo:session): session closed for user root 2024-04-26T21:27:59.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:59 smithi142 ceph-mon[32944]: pgmap v82: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:27:59.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:59 smithi142 ceph-mon[32944]: Detected new or changed devices on smithi142 2024-04-26T21:27:59.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:59 smithi142 ceph-mon[32944]: Adjusting osd_memory_target on smithi142 to 17088M 2024-04-26T21:27:59.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:59 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T21:27:59.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:59 smithi142 ceph-mon[32944]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T21:27:59.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:59 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T21:27:59.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:59 smithi050 ceph-mon[32753]: pgmap v82: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:27:59.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:59 smithi050 ceph-mon[32753]: Detected new or changed devices on smithi142 2024-04-26T21:27:59.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:59 smithi050 ceph-mon[32753]: Adjusting osd_memory_target on smithi142 to 17088M 2024-04-26T21:27:59.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:59 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T21:27:59.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:59 smithi050 ceph-mon[32753]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T21:27:59.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:59 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T21:27:59.869 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 21:27:59 smithi050 sudo[41483]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T21:27:59.869 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 21:27:59 smithi050 sudo[41483]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T21:27:59.869 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 21:27:59 smithi050 sudo[41483]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T21:27:59.869 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 21:27:59 smithi050 sudo[41483]: pam_unix(sudo:session): session closed for user root 2024-04-26T21:27:59.869 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 21:27:59 smithi050 sudo[41486]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T21:27:59.869 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 21:27:59 smithi050 sudo[41486]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T21:27:59.869 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 21:27:59 smithi050 sudo[41486]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T21:27:59.869 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 21:27:59 smithi050 sudo[41486]: pam_unix(sudo:session): session closed for user root 2024-04-26T21:28:00.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:59 smithi046 sudo[48291]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T21:28:00.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:59 smithi046 sudo[48291]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T21:28:00.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:59 smithi046 sudo[48291]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T21:28:00.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:59 smithi046 sudo[48291]: pam_unix(sudo:session): session closed for user root 2024-04-26T21:28:00.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:59 smithi046 sudo[48294]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T21:28:00.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:59 smithi046 sudo[48294]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T21:28:00.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:59 smithi046 sudo[48294]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T21:28:00.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:27:59 smithi046 sudo[48294]: pam_unix(sudo:session): session closed for user root 2024-04-26T21:28:00.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.164+0000 7f9449239700 1 -- 172.21.15.46:0/3434270312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9444103f40 msgr2=0x7f944410a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:00.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.164+0000 7f9449239700 1 --2- 172.21.15.46:0/3434270312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9444103f40 0x7f944410a7f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f943c00b0d0 tx=0x7f943c00b3e0 comp rx=0 tx=0).stop 2024-04-26T21:28:00.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.164+0000 7f9449239700 1 -- 172.21.15.46:0/3434270312 shutdown_connections 2024-04-26T21:28:00.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.164+0000 7f9449239700 1 --2- 172.21.15.46:0/3434270312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9444103f40 0x7f944410a7f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:00.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.164+0000 7f9449239700 1 --2- 172.21.15.46:0/3434270312 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f94441023a0 0x7f94441027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:00.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.164+0000 7f9449239700 1 --2- 172.21.15.46:0/3434270312 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94441035a0 0x7f9444103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:00.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.164+0000 7f9449239700 1 -- 172.21.15.46:0/3434270312 >> 172.21.15.46:0/3434270312 conn(0x7f94440fd940 msgr2=0x7f94440ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:00.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.164+0000 7f9449239700 1 -- 172.21.15.46:0/3434270312 shutdown_connections 2024-04-26T21:28:00.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.164+0000 7f9449239700 1 -- 172.21.15.46:0/3434270312 wait complete. 2024-04-26T21:28:00.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.165+0000 7f9449239700 1 Processor -- start 2024-04-26T21:28:00.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.165+0000 7f9449239700 1 -- start start 2024-04-26T21:28:00.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.166+0000 7f9449239700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94441023a0 0x7f9444198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:00.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.166+0000 7f9449239700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94441035a0 0x7f94441988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:00.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.166+0000 7f9449239700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9444103f40 0x7f944419f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:00.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.166+0000 7f9449239700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f944410de10 con 0x7f94441035a0 2024-04-26T21:28:00.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.166+0000 7f9449239700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f944410dc90 con 0x7f94441023a0 2024-04-26T21:28:00.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.166+0000 7f9449239700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f944410df90 con 0x7f9444103f40 2024-04-26T21:28:00.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.166+0000 7f944259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94441035a0 0x7f94441988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:00.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.166+0000 7f944259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94441035a0 0x7f94441988d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49706/0 (socket says 172.21.15.46:49706) 2024-04-26T21:28:00.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.166+0000 7f944259c700 1 -- 172.21.15.46:0/2870716026 learned_addr learned my addr 172.21.15.46:0/2870716026 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:28:00.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.166+0000 7f944359e700 1 --2- 172.21.15.46:0/2870716026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9444103f40 0x7f944419f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:00.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.166+0000 7f9442d9d700 1 --2- 172.21.15.46:0/2870716026 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94441023a0 0x7f9444198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:00.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.166+0000 7f944259c700 1 -- 172.21.15.46:0/2870716026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9444103f40 msgr2=0x7f944419f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:00.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.166+0000 7f944259c700 1 --2- 172.21.15.46:0/2870716026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9444103f40 0x7f944419f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:00.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.166+0000 7f944259c700 1 -- 172.21.15.46:0/2870716026 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94441023a0 msgr2=0x7f9444198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:00.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.166+0000 7f944259c700 1 --2- 172.21.15.46:0/2870716026 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94441023a0 0x7f9444198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:00.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.166+0000 7f944259c700 1 -- 172.21.15.46:0/2870716026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f944406bb50 con 0x7f94441035a0 2024-04-26T21:28:00.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.166+0000 7f944259c700 1 --2- 172.21.15.46:0/2870716026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94441035a0 0x7f94441988d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f943800c9e0 tx=0x7f943800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:00.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.166+0000 7f942bfff700 1 -- 172.21.15.46:0/2870716026 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9438007ac0 con 0x7f94441035a0 2024-04-26T21:28:00.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.166+0000 7f9449239700 1 -- 172.21.15.46:0/2870716026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f944406be40 con 0x7f94441035a0 2024-04-26T21:28:00.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.167+0000 7f942bfff700 1 -- 172.21.15.46:0/2870716026 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9438007c60 con 0x7f94441035a0 2024-04-26T21:28:00.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.167+0000 7f942bfff700 1 -- 172.21.15.46:0/2870716026 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9438018660 con 0x7f94441035a0 2024-04-26T21:28:00.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.167+0000 7f9449239700 1 -- 172.21.15.46:0/2870716026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f944406c3d0 con 0x7f94441035a0 2024-04-26T21:28:00.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.168+0000 7f9449239700 1 -- 172.21.15.46:0/2870716026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f944406a110 con 0x7f94441035a0 2024-04-26T21:28:00.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.169+0000 7f942bfff700 1 -- 172.21.15.46:0/2870716026 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f943800f450 con 0x7f94441035a0 2024-04-26T21:28:00.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.170+0000 7f942bfff700 1 --2- 172.21.15.46:0/2870716026 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f942c0a3bb0 0x7f942c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:00.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.170+0000 7f9442d9d700 1 --2- 172.21.15.46:0/2870716026 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f942c0a3bb0 0x7f942c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:00.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.170+0000 7f942bfff700 1 -- 172.21.15.46:0/2870716026 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3237+0+0 (secure 0 0 0) 0x7f94380cf110 con 0x7f94441035a0 2024-04-26T21:28:00.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.170+0000 7f9442d9d700 1 --2- 172.21.15.46:0/2870716026 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f942c0a3bb0 0x7f942c0a6070 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9444103400 tx=0x7f943400b5b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:00.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.174+0000 7f942bfff700 1 -- 172.21.15.46:0/2870716026 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9438014030 con 0x7f94441035a0 2024-04-26T21:28:00.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:59 smithi142 sudo[41988]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T21:28:00.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:59 smithi142 sudo[41988]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T21:28:00.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:59 smithi142 sudo[41988]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T21:28:00.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:59 smithi142 sudo[41988]: pam_unix(sudo:session): session closed for user root 2024-04-26T21:28:00.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:59 smithi142 sudo[41991]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T21:28:00.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:59 smithi142 sudo[41991]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T21:28:00.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:59 smithi142 sudo[41991]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T21:28:00.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:27:59 smithi142 sudo[41991]: pam_unix(sudo:session): session closed for user root 2024-04-26T21:28:00.291 INFO:journalctl@ceph.osd.2.smithi142.stdout:Apr 26 21:27:59 smithi142 sudo[41982]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T21:28:00.292 INFO:journalctl@ceph.osd.2.smithi142.stdout:Apr 26 21:27:59 smithi142 sudo[41982]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T21:28:00.292 INFO:journalctl@ceph.osd.2.smithi142.stdout:Apr 26 21:27:59 smithi142 sudo[41982]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T21:28:00.292 INFO:journalctl@ceph.osd.2.smithi142.stdout:Apr 26 21:27:59 smithi142 sudo[41982]: pam_unix(sudo:session): session closed for user root 2024-04-26T21:28:00.292 INFO:journalctl@ceph.osd.2.smithi142.stdout:Apr 26 21:27:59 smithi142 sudo[41985]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T21:28:00.292 INFO:journalctl@ceph.osd.2.smithi142.stdout:Apr 26 21:27:59 smithi142 sudo[41985]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T21:28:00.292 INFO:journalctl@ceph.osd.2.smithi142.stdout:Apr 26 21:27:59 smithi142 sudo[41985]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T21:28:00.292 INFO:journalctl@ceph.osd.2.smithi142.stdout:Apr 26 21:27:59 smithi142 sudo[41985]: pam_unix(sudo:session): session closed for user root 2024-04-26T21:28:00.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:59 smithi050 sudo[41489]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T21:28:00.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:59 smithi050 sudo[41489]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T21:28:00.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:59 smithi050 sudo[41489]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T21:28:00.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:59 smithi050 sudo[41489]: pam_unix(sudo:session): session closed for user root 2024-04-26T21:28:00.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:59 smithi050 sudo[41492]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T21:28:00.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:59 smithi050 sudo[41492]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T21:28:00.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:59 smithi050 sudo[41492]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T21:28:00.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:27:59 smithi050 sudo[41492]: pam_unix(sudo:session): session closed for user root 2024-04-26T21:28:00.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.391+0000 7f9449239700 1 -- 172.21.15.46:0/2870716026 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f9444063860 con 0x7f942c0a3bb0 2024-04-26T21:28:00.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:00 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T21:28:00.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:00 smithi046 ceph-mon[29201]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T21:28:00.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:00 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:00.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:00 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:00.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:00 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:00.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:00 smithi046 ceph-mon[29201]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T21:28:00.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:00 smithi046 ceph-mon[29201]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T21:28:00.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:00 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:28:00.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:00 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:28:00.805 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:00 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:28:00.805 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:00 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:28:00.805 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:00 smithi046 ceph-mon[29201]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T21:28:00.805 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:00 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:28:00.805 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:00 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:28:00.805 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:00 smithi046 ceph-mon[29201]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T21:28:00.805 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:00 smithi046 ceph-mon[29201]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T21:28:00.805 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:00 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:28:00.805 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:00 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:28:00.805 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:00 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:28:00.805 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:00 smithi046 ceph-mon[29201]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T21:28:00.805 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:00 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:28:00.805 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:00 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:28:00.805 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:00 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:00.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:00.852+0000 7f942bfff700 1 -- 172.21.15.46:0/2870716026 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9438098f80 con 0x7f94441035a0 2024-04-26T21:28:00.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:00 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T21:28:00.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:00 smithi050 ceph-mon[32753]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T21:28:00.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:00 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:00.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:00 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:00.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:00 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:00.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:00 smithi050 ceph-mon[32753]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T21:28:00.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:00 smithi050 ceph-mon[32753]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T21:28:00.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:00 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:28:00.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:00 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:28:00.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:00 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:28:00.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:00 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:28:00.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:00 smithi050 ceph-mon[32753]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T21:28:00.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:00 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:28:00.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:00 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:28:00.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:00 smithi050 ceph-mon[32753]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T21:28:00.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:00 smithi050 ceph-mon[32753]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T21:28:00.870 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:00 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:28:00.870 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:00 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:28:00.870 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:00 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:28:00.870 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:00 smithi050 ceph-mon[32753]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T21:28:00.870 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:00 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:28:00.870 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:00 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:28:00.870 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:00 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:01.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:00 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T21:28:01.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:00 smithi142 ceph-mon[32944]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T21:28:01.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:00 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:01.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:00 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:01.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:00 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:01.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:00 smithi142 ceph-mon[32944]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T21:28:01.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:00 smithi142 ceph-mon[32944]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T21:28:01.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:00 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:28:01.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:00 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:28:01.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:00 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:28:01.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:00 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:28:01.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:00 smithi142 ceph-mon[32944]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T21:28:01.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:00 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:28:01.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:00 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:28:01.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:00 smithi142 ceph-mon[32944]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T21:28:01.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:00 smithi142 ceph-mon[32944]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T21:28:01.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:00 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:28:01.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:00 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:28:01.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:00 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:28:01.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:00 smithi142 ceph-mon[32944]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T21:28:01.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:00 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:28:01.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:00 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:28:01.043 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:00 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:01.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:01 smithi046 ceph-mon[29201]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:01.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:01 smithi046 ceph-mon[29201]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:28:01.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:01 smithi046 ceph-mon[29201]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T21:28:01.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:01 smithi046 ceph-mon[29201]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T21:28:01.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:01 smithi050 ceph-mon[32753]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:01.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:01 smithi050 ceph-mon[32753]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:28:01.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:01 smithi050 ceph-mon[32753]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T21:28:01.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:01 smithi050 ceph-mon[32753]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T21:28:02.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:01 smithi142 ceph-mon[32944]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:02.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:01 smithi142 ceph-mon[32944]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:28:02.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:01 smithi142 ceph-mon[32944]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T21:28:02.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:01 smithi142 ceph-mon[32944]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T21:28:03.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:03 smithi046 ceph-mon[29201]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:03.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:03 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2759143616' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34d2401f-5fcc-491c-8fcf-c3c492611f23"}]: dispatch 2024-04-26T21:28:03.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:03 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2759143616' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34d2401f-5fcc-491c-8fcf-c3c492611f23"}]': finished 2024-04-26T21:28:03.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:03 smithi046 ceph-mon[29201]: osdmap e22: 4 total, 3 up, 4 in 2024-04-26T21:28:03.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:03 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:28:03.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:03 smithi050 ceph-mon[32753]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:03.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:03 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/2759143616' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34d2401f-5fcc-491c-8fcf-c3c492611f23"}]: dispatch 2024-04-26T21:28:03.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:03 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/2759143616' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34d2401f-5fcc-491c-8fcf-c3c492611f23"}]': finished 2024-04-26T21:28:03.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:03 smithi050 ceph-mon[32753]: osdmap e22: 4 total, 3 up, 4 in 2024-04-26T21:28:03.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:03 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:28:04.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:03 smithi142 ceph-mon[32944]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:04.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:03 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/2759143616' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "34d2401f-5fcc-491c-8fcf-c3c492611f23"}]: dispatch 2024-04-26T21:28:04.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:03 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/2759143616' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "34d2401f-5fcc-491c-8fcf-c3c492611f23"}]': finished 2024-04-26T21:28:04.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:03 smithi142 ceph-mon[32944]: osdmap e22: 4 total, 3 up, 4 in 2024-04-26T21:28:04.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:03 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:28:04.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:04 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/954555489' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:28:05.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:04 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/954555489' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:28:05.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:04 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/954555489' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:28:05.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:05 smithi050 ceph-mon[32753]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:06.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:05 smithi142 ceph-mon[32944]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:06.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:05 smithi046 ceph-mon[29201]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:07.840 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:07 smithi046 ceph-mon[29201]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:07.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:07 smithi050 ceph-mon[32753]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:08.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:07 smithi142 ceph-mon[32944]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:09.662 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:09 smithi046 ceph-mon[29201]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:09.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:09 smithi050 ceph-mon[32753]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:10.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:09 smithi142 ceph-mon[32944]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:11.843 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:11 smithi142 ceph-mon[32944]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:11.843 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:11 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T21:28:11.843 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:11 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:11.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:11 smithi050 ceph-mon[32753]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:11.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:11 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T21:28:11.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:11 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:11.916 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:11 smithi046 ceph-mon[29201]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:11.916 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:11 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T21:28:11.916 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:11 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:12.820 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:12 smithi046 ceph-mon[29201]: Deploying daemon osd.3 on smithi046 2024-04-26T21:28:12.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:12 smithi050 ceph-mon[32753]: Deploying daemon osd.3 on smithi046 2024-04-26T21:28:13.035 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:12 smithi142 ceph-mon[32944]: Deploying daemon osd.3 on smithi046 2024-04-26T21:28:13.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:13 smithi046 ceph-mon[29201]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:13.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:13 smithi050 ceph-mon[32753]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:14.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:13 smithi142 ceph-mon[32944]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:15.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:15 smithi050 ceph-mon[32753]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:15.949 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:15 smithi046 ceph-mon[29201]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:16.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:15 smithi142 ceph-mon[32944]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:17.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:17 smithi046 ceph-mon[29201]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:17.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:17 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:17.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:17 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:17.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:17 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:28:17.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:17 smithi050 ceph-mon[32753]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:17.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:17 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:17.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:17 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:17.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:17 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:28:18.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:17 smithi142 ceph-mon[32944]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:18.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:17 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:18.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:17 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:18.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:17 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:28:19.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:19 smithi046 ceph-mon[29201]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:19.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:19 smithi050 ceph-mon[32753]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:20.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:19 smithi142 ceph-mon[32944]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:20.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:20.203+0000 7f942bfff700 1 -- 172.21.15.46:0/2870716026 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9444063860 con 0x7f942c0a3bb0 2024-04-26T21:28:20.204 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 3 on host 'smithi046' 2024-04-26T21:28:20.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:20.205+0000 7f9449239700 1 -- 172.21.15.46:0/2870716026 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f942c0a3bb0 msgr2=0x7f942c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:20.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:20.205+0000 7f9449239700 1 --2- 172.21.15.46:0/2870716026 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f942c0a3bb0 0x7f942c0a6070 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9444103400 tx=0x7f943400b5b0 comp rx=0 tx=0).stop 2024-04-26T21:28:20.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:20.205+0000 7f9449239700 1 -- 172.21.15.46:0/2870716026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94441035a0 msgr2=0x7f94441988d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:20.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:20.205+0000 7f9449239700 1 --2- 172.21.15.46:0/2870716026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94441035a0 0x7f94441988d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f943800c9e0 tx=0x7f943800ccf0 comp rx=0 tx=0).stop 2024-04-26T21:28:20.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:20.205+0000 7f9449239700 1 -- 172.21.15.46:0/2870716026 shutdown_connections 2024-04-26T21:28:20.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:20.205+0000 7f9449239700 1 --2- 172.21.15.46:0/2870716026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9444103f40 0x7f944419f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:20.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:20.205+0000 7f9449239700 1 --2- 172.21.15.46:0/2870716026 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f94441023a0 0x7f9444198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:20.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:20.205+0000 7f9449239700 1 --2- 172.21.15.46:0/2870716026 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f942c0a3bb0 0x7f942c0a6070 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:20.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:20.205+0000 7f9449239700 1 --2- 172.21.15.46:0/2870716026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94441035a0 0x7f94441988d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:20.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:20.205+0000 7f9449239700 1 -- 172.21.15.46:0/2870716026 >> 172.21.15.46:0/2870716026 conn(0x7f94440fd940 msgr2=0x7f9444106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:20.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:20.205+0000 7f9449239700 1 -- 172.21.15.46:0/2870716026 shutdown_connections 2024-04-26T21:28:20.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:28:20.205+0000 7f9449239700 1 -- 172.21.15.46:0/2870716026 wait complete. 2024-04-26T21:28:20.943 DEBUG:teuthology.orchestra.run.smithi046:osd.3> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.3.service 2024-04-26T21:28:20.945 INFO:tasks.cephadm:Deploying osd.4 on smithi050 with /dev/vg_nvme/lv_3... 2024-04-26T21:28:20.945 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T21:28:21.199 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T21:28:21.321 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:21 smithi050 ceph-mon[32753]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:21.321 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:21 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:21.322 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:21 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:21.322 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:21 smithi050 ceph-mon[32753]: from='osd.3 [v2:172.21.15.46:6810/2435181760,v1:172.21.15.46:6811/2435181760]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T21:28:21.322 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:21 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:21.322 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:21 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:21.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:21 smithi142 ceph-mon[32944]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:21.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:21 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:21.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:21 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:21.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:21 smithi142 ceph-mon[32944]: from='osd.3 [v2:172.21.15.46:6810/2435181760,v1:172.21.15.46:6811/2435181760]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T21:28:21.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:21 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:21.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:21 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:21.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:21 smithi046 ceph-mon[29201]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:21.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:21 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:21.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:21 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:21.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:21 smithi046 ceph-mon[29201]: from='osd.3 [v2:172.21.15.46:6810/2435181760,v1:172.21.15.46:6811/2435181760]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T21:28:21.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:21 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:21.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:21 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:22.752 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:22 smithi050 ceph-mon[32753]: from='osd.3 [v2:172.21.15.46:6810/2435181760,v1:172.21.15.46:6811/2435181760]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T21:28:22.752 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:22 smithi050 ceph-mon[32753]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T21:28:22.753 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:22 smithi050 ceph-mon[32753]: from='osd.3 [v2:172.21.15.46:6810/2435181760,v1:172.21.15.46:6811/2435181760]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-26T21:28:22.753 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:22 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:28:22.770 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:22 smithi046 ceph-mon[29201]: from='osd.3 [v2:172.21.15.46:6810/2435181760,v1:172.21.15.46:6811/2435181760]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T21:28:22.770 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:22 smithi046 ceph-mon[29201]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T21:28:22.770 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:22 smithi046 ceph-mon[29201]: from='osd.3 [v2:172.21.15.46:6810/2435181760,v1:172.21.15.46:6811/2435181760]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-26T21:28:22.770 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:22 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:28:22.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:22 smithi142 ceph-mon[32944]: from='osd.3 [v2:172.21.15.46:6810/2435181760,v1:172.21.15.46:6811/2435181760]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T21:28:22.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:22 smithi142 ceph-mon[32944]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T21:28:22.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:22 smithi142 ceph-mon[32944]: from='osd.3 [v2:172.21.15.46:6810/2435181760,v1:172.21.15.46:6811/2435181760]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-26T21:28:22.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:22 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:28:23.357 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T21:28:23.358 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T21:28:23.358 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10+0 records in 2024-04-26T21:28:23.358 INFO:teuthology.orchestra.run.smithi050.stderr:10+0 records out 2024-04-26T21:28:23.358 INFO:teuthology.orchestra.run.smithi050.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00871118 s, 1.2 GB/s 2024-04-26T21:28:23.358 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 2024-04-26T21:28:23.358 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping successful for: 2024-04-26T21:28:23.554 INFO:journalctl@ceph.osd.3.smithi046.stdout:Apr 26 21:28:23 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-3[50586]: 2024-04-26T21:28:23.055+0000 7f13c12b6700 -1 osd.3 0 waiting for initial osdmap 2024-04-26T21:28:23.554 INFO:journalctl@ceph.osd.3.smithi046.stdout:Apr 26 21:28:23 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-3[50586]: 2024-04-26T21:28:23.061+0000 7f13b90a1700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T21:28:23.685 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:23 smithi050 ceph-mon[32753]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:23.685 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:23 smithi050 ceph-mon[32753]: from='osd.3 [v2:172.21.15.46:6810/2435181760,v1:172.21.15.46:6811/2435181760]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-26T21:28:23.685 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:23 smithi050 ceph-mon[32753]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T21:28:23.685 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:23 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:28:23.685 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:23 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:28:23.685 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:23 smithi050 ceph-mon[32753]: from='osd.3 [v2:172.21.15.46:6810/2435181760,v1:172.21.15.46:6811/2435181760]' entity='osd.3' 2024-04-26T21:28:23.859 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:23 smithi046 ceph-mon[29201]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:23.859 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:23 smithi046 ceph-mon[29201]: from='osd.3 [v2:172.21.15.46:6810/2435181760,v1:172.21.15.46:6811/2435181760]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-26T21:28:23.859 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:23 smithi046 ceph-mon[29201]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T21:28:23.859 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:23 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:28:23.859 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:23 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:28:23.860 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:23 smithi046 ceph-mon[29201]: from='osd.3 [v2:172.21.15.46:6810/2435181760,v1:172.21.15.46:6811/2435181760]' entity='osd.3' 2024-04-26T21:28:24.003 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi050:vg_nvme/lv_3 2024-04-26T21:28:24.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:23 smithi142 ceph-mon[32944]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:28:24.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:23 smithi142 ceph-mon[32944]: from='osd.3 [v2:172.21.15.46:6810/2435181760,v1:172.21.15.46:6811/2435181760]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-26T21:28:24.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:23 smithi142 ceph-mon[32944]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T21:28:24.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:23 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:28:24.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:23 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:28:24.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:23 smithi142 ceph-mon[32944]: from='osd.3 [v2:172.21.15.46:6810/2435181760,v1:172.21.15.46:6811/2435181760]' entity='osd.3' 2024-04-26T21:28:24.252 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T21:28:24.627 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:24 smithi050 ceph-mon[32753]: purged_snaps scrub starts 2024-04-26T21:28:24.628 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:24 smithi050 ceph-mon[32753]: purged_snaps scrub ok 2024-04-26T21:28:24.628 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:24 smithi050 ceph-mon[32753]: osd.3 [v2:172.21.15.46:6810/2435181760,v1:172.21.15.46:6811/2435181760] boot 2024-04-26T21:28:24.628 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:24 smithi050 ceph-mon[32753]: osdmap e25: 4 total, 4 up, 4 in 2024-04-26T21:28:24.628 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:24 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:28:24.628 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:24 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:24.628 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:24 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:24.628 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:24 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:28:24.628 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:24 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:28:24.628 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:24 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:24.628 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:24 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:24.628 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:24 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:28:24.628 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:24 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:24.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:24 smithi046 ceph-mon[29201]: purged_snaps scrub starts 2024-04-26T21:28:24.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:24 smithi046 ceph-mon[29201]: purged_snaps scrub ok 2024-04-26T21:28:24.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:24 smithi046 ceph-mon[29201]: osd.3 [v2:172.21.15.46:6810/2435181760,v1:172.21.15.46:6811/2435181760] boot 2024-04-26T21:28:24.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:24 smithi046 ceph-mon[29201]: osdmap e25: 4 total, 4 up, 4 in 2024-04-26T21:28:24.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:24 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:28:24.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:24 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:24.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:24 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:24.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:24 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:28:24.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:24 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:28:24.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:24 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:24.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:24 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:24.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:24 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:28:24.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:24 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:24 smithi142 ceph-mon[32944]: purged_snaps scrub starts 2024-04-26T21:28:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:24 smithi142 ceph-mon[32944]: purged_snaps scrub ok 2024-04-26T21:28:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:24 smithi142 ceph-mon[32944]: osd.3 [v2:172.21.15.46:6810/2435181760,v1:172.21.15.46:6811/2435181760] boot 2024-04-26T21:28:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:24 smithi142 ceph-mon[32944]: osdmap e25: 4 total, 4 up, 4 in 2024-04-26T21:28:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:24 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:28:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:24 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:24 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:24 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:28:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:24 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:28:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:24 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:24 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:24 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:28:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:24 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:25.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:25 smithi050 ceph-mon[32753]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:28:25.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:25 smithi050 ceph-mon[32753]: Detected new or changed devices on smithi046 2024-04-26T21:28:25.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:25 smithi050 ceph-mon[32753]: Adjusting osd_memory_target on smithi046 to 8546M 2024-04-26T21:28:25.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:25 smithi050 ceph-mon[32753]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T21:28:25.655 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:25 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:28:25.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:25 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:25.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:25 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:28:25.656 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:25 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:25.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.901+0000 7fe07b534700 1 -- 172.21.15.50:0/3721112356 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe074102380 msgr2=0x7fe0741027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:25.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.901+0000 7fe07b534700 1 --2- 172.21.15.50:0/3721112356 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe074102380 0x7fe0741027a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe0680099c0 tx=0x7fe068009cd0 comp rx=0 tx=0).stop 2024-04-26T21:28:25.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.901+0000 7fe07b534700 1 -- 172.21.15.50:0/3721112356 shutdown_connections 2024-04-26T21:28:25.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.901+0000 7fe07b534700 1 --2- 172.21.15.50:0/3721112356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe074103f20 0x7fe07410a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:25.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.901+0000 7fe07b534700 1 --2- 172.21.15.50:0/3721112356 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe074102380 0x7fe0741027a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:25.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.901+0000 7fe07b534700 1 --2- 172.21.15.50:0/3721112356 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe074103580 0x7fe0741039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:25.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.901+0000 7fe07b534700 1 -- 172.21.15.50:0/3721112356 >> 172.21.15.50:0/3721112356 conn(0x7fe0740fd920 msgr2=0x7fe0740ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:25.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.901+0000 7fe07b534700 1 -- 172.21.15.50:0/3721112356 shutdown_connections 2024-04-26T21:28:25.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.901+0000 7fe07b534700 1 -- 172.21.15.50:0/3721112356 wait complete. 2024-04-26T21:28:25.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.902+0000 7fe07b534700 1 Processor -- start 2024-04-26T21:28:25.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.902+0000 7fe07b534700 1 -- start start 2024-04-26T21:28:25.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.902+0000 7fe07b534700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe074102380 0x7fe074199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:25.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.902+0000 7fe07b534700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe074103580 0x7fe074199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:25.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.902+0000 7fe07b534700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe074103f20 0x7fe0741a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:25.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.902+0000 7fe07b534700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe07410de10 con 0x7fe074103580 2024-04-26T21:28:25.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.902+0000 7fe07b534700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe07410dc90 con 0x7fe074103f20 2024-04-26T21:28:25.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.902+0000 7fe07b534700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe07410df90 con 0x7fe074102380 2024-04-26T21:28:25.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.902+0000 7fe079ad1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe074103f20 0x7fe0741a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:25.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.902+0000 7fe079ad1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe074103f20 0x7fe0741a0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51618/0 (socket says 172.21.15.50:51618) 2024-04-26T21:28:25.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.902+0000 7fe079ad1700 1 -- 172.21.15.50:0/837651623 learned_addr learned my addr 172.21.15.50:0/837651623 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T21:28:25.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.902+0000 7fe0792d0700 1 --2- 172.21.15.50:0/837651623 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe074102380 0x7fe074199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:25.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.902+0000 7fe078acf700 1 --2- 172.21.15.50:0/837651623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe074103580 0x7fe074199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:25.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.903+0000 7fe079ad1700 1 -- 172.21.15.50:0/837651623 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe074102380 msgr2=0x7fe074199690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:25.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.903+0000 7fe079ad1700 1 --2- 172.21.15.50:0/837651623 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe074102380 0x7fe074199690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:25.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.903+0000 7fe079ad1700 1 -- 172.21.15.50:0/837651623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe074103580 msgr2=0x7fe074199bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:25.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.903+0000 7fe079ad1700 1 --2- 172.21.15.50:0/837651623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe074103580 0x7fe074199bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:25.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.903+0000 7fe079ad1700 1 -- 172.21.15.50:0/837651623 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0741a12a0 con 0x7fe074103f20 2024-04-26T21:28:25.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.903+0000 7fe079ad1700 1 --2- 172.21.15.50:0/837651623 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe074103f20 0x7fe0741a0c00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe06c00c370 tx=0x7fe06c00c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:25.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.903+0000 7fe0667fc700 1 -- 172.21.15.50:0/837651623 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe06c0092e0 con 0x7fe074103f20 2024-04-26T21:28:25.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.903+0000 7fe07b534700 1 -- 172.21.15.50:0/837651623 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0741a1530 con 0x7fe074103f20 2024-04-26T21:28:25.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.903+0000 7fe0667fc700 1 -- 172.21.15.50:0/837651623 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe06c003a00 con 0x7fe074103f20 2024-04-26T21:28:25.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.903+0000 7fe0667fc700 1 -- 172.21.15.50:0/837651623 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe06c0046e0 con 0x7fe074103f20 2024-04-26T21:28:25.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.903+0000 7fe07b534700 1 -- 172.21.15.50:0/837651623 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0741a1a70 con 0x7fe074103f20 2024-04-26T21:28:25.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.904+0000 7fe07b534700 1 -- 172.21.15.50:0/837651623 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe07406a110 con 0x7fe074103f20 2024-04-26T21:28:25.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.905+0000 7fe0667fc700 1 -- 172.21.15.50:0/837651623 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe06c01a020 con 0x7fe074103f20 2024-04-26T21:28:25.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.906+0000 7fe0667fc700 1 --2- 172.21.15.50:0/837651623 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe0600a3c40 0x7fe0600a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:25.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.906+0000 7fe0792d0700 1 --2- 172.21.15.50:0/837651623 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe0600a3c40 0x7fe0600a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:25.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.906+0000 7fe0667fc700 1 -- 172.21.15.50:0/837651623 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3698+0+0 (secure 0 0 0) 0x7fe06c0cea40 con 0x7fe074103f20 2024-04-26T21:28:25.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.906+0000 7fe0792d0700 1 --2- 172.21.15.50:0/837651623 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe0600a3c40 0x7fe0600a6100 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe068000c00 tx=0x7fe068005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:25.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:25.911+0000 7fe0667fc700 1 -- 172.21.15.50:0/837651623 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe06c010040 con 0x7fe074103f20 2024-04-26T21:28:26.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:25 smithi142 ceph-mon[32944]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:28:26.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:25 smithi142 ceph-mon[32944]: Detected new or changed devices on smithi046 2024-04-26T21:28:26.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:25 smithi142 ceph-mon[32944]: Adjusting osd_memory_target on smithi046 to 8546M 2024-04-26T21:28:26.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:25 smithi142 ceph-mon[32944]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T21:28:26.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:25 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:28:26.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:25 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:26.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:25 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:28:26.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:25 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:26.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:25 smithi046 ceph-mon[29201]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:28:26.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:25 smithi046 ceph-mon[29201]: Detected new or changed devices on smithi046 2024-04-26T21:28:26.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:25 smithi046 ceph-mon[29201]: Adjusting osd_memory_target on smithi046 to 8546M 2024-04-26T21:28:26.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:25 smithi046 ceph-mon[29201]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T21:28:26.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:25 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:28:26.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:25 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:26.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:25 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:28:26.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:25 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:26.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:26.127+0000 7fe07b534700 1 -- 172.21.15.50:0/837651623 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fe07406b810 con 0x7fe0600a3c40 2024-04-26T21:28:26.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:26 smithi050 ceph-mon[32753]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T21:28:26.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:28:26.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:28:26.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:27.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:26 smithi142 ceph-mon[32944]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T21:28:27.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:26 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:28:27.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:26 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:28:27.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:26 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:27.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:26 smithi046 ceph-mon[29201]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T21:28:27.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:26 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:28:27.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:26 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:28:27.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:26 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:27.885 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:27 smithi050 ceph-mon[32753]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 142 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:28:27.886 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:27 smithi050 ceph-mon[32753]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:28:27.886 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:27 smithi050 ceph-mon[32753]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T21:28:28.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:27 smithi142 ceph-mon[32944]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 142 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:28:28.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:27 smithi142 ceph-mon[32944]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:28:28.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:27 smithi142 ceph-mon[32944]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T21:28:28.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:27 smithi046 ceph-mon[29201]: pgmap v105: 1 pgs: 1 remapped+peering; 577 KiB data, 142 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:28:28.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:27 smithi046 ceph-mon[29201]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:28:28.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:27 smithi046 ceph-mon[29201]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T21:28:29.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:28 smithi142 ceph-mon[32944]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T21:28:29.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:28 smithi046 ceph-mon[29201]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T21:28:29.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:28 smithi050 ceph-mon[32753]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T21:28:30.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:29 smithi142 ceph-mon[32944]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 142 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:28:30.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:29 smithi142 ceph-mon[32944]: from='client.? 172.21.15.50:0/2599793283' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "986bb709-e4cf-400d-982e-c3f5ac413e0e"}]: dispatch 2024-04-26T21:28:30.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:29 smithi142 ceph-mon[32944]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "986bb709-e4cf-400d-982e-c3f5ac413e0e"}]: dispatch 2024-04-26T21:28:30.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:29 smithi142 ceph-mon[32944]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "986bb709-e4cf-400d-982e-c3f5ac413e0e"}]': finished 2024-04-26T21:28:30.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:29 smithi142 ceph-mon[32944]: osdmap e30: 5 total, 4 up, 5 in 2024-04-26T21:28:30.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:29 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:28:30.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:29 smithi142 ceph-mon[32944]: from='client.? 172.21.15.50:0/3381992383' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:28:30.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:29 smithi046 ceph-mon[29201]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 142 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:28:30.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:29 smithi046 ceph-mon[29201]: from='client.? 172.21.15.50:0/2599793283' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "986bb709-e4cf-400d-982e-c3f5ac413e0e"}]: dispatch 2024-04-26T21:28:30.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:29 smithi046 ceph-mon[29201]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "986bb709-e4cf-400d-982e-c3f5ac413e0e"}]: dispatch 2024-04-26T21:28:30.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:29 smithi046 ceph-mon[29201]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "986bb709-e4cf-400d-982e-c3f5ac413e0e"}]': finished 2024-04-26T21:28:30.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:29 smithi046 ceph-mon[29201]: osdmap e30: 5 total, 4 up, 5 in 2024-04-26T21:28:30.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:29 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:28:30.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:29 smithi046 ceph-mon[29201]: from='client.? 172.21.15.50:0/3381992383' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:28:30.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:29 smithi050 ceph-mon[32753]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 142 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:28:30.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:29 smithi050 ceph-mon[32753]: from='client.? 172.21.15.50:0/2599793283' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "986bb709-e4cf-400d-982e-c3f5ac413e0e"}]: dispatch 2024-04-26T21:28:30.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:29 smithi050 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "986bb709-e4cf-400d-982e-c3f5ac413e0e"}]: dispatch 2024-04-26T21:28:30.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:29 smithi050 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "986bb709-e4cf-400d-982e-c3f5ac413e0e"}]': finished 2024-04-26T21:28:30.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:29 smithi050 ceph-mon[32753]: osdmap e30: 5 total, 4 up, 5 in 2024-04-26T21:28:30.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:29 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:28:30.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:29 smithi050 ceph-mon[32753]: from='client.? 172.21.15.50:0/3381992383' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:28:32.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:31 smithi142 ceph-mon[32944]: pgmap v110: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:28:32.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:31 smithi046 ceph-mon[29201]: pgmap v110: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:28:32.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:31 smithi050 ceph-mon[32753]: pgmap v110: 1 pgs: 1 remapped+peering; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:28:33.921 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:33 smithi050 ceph-mon[32753]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T21:28:34.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:33 smithi142 ceph-mon[32944]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T21:28:34.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:33 smithi046 ceph-mon[29201]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T21:28:35.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:35 smithi050 ceph-mon[32753]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-26T21:28:36.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:35 smithi142 ceph-mon[32944]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-26T21:28:36.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:35 smithi046 ceph-mon[29201]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-26T21:28:36.976 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:36 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T21:28:36.976 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:36 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:37.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:36 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T21:28:37.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:36 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:37.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:36 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T21:28:37.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:36 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:37.763 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:37 smithi050 ceph-mon[32753]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-26T21:28:37.764 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:37 smithi050 ceph-mon[32753]: Deploying daemon osd.4 on smithi050 2024-04-26T21:28:38.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:37 smithi142 ceph-mon[32944]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-26T21:28:38.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:37 smithi142 ceph-mon[32944]: Deploying daemon osd.4 on smithi050 2024-04-26T21:28:38.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:37 smithi046 ceph-mon[29201]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-26T21:28:38.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:37 smithi046 ceph-mon[29201]: Deploying daemon osd.4 on smithi050 2024-04-26T21:28:39.925 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:39 smithi050 ceph-mon[32753]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T21:28:40.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:39 smithi142 ceph-mon[32944]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T21:28:40.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:39 smithi046 ceph-mon[29201]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T21:28:42.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:41 smithi142 ceph-mon[32944]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-26T21:28:42.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:41 smithi046 ceph-mon[29201]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-26T21:28:42.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:41 smithi050 ceph-mon[32753]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-26T21:28:43.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:42 smithi142 ceph-mon[32944]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-26T21:28:43.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:42 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:43.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:42 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:43.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:42 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:28:43.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:42 smithi046 ceph-mon[29201]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-26T21:28:43.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:42 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:43.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:42 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:43.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:42 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:28:43.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:42 smithi050 ceph-mon[32753]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-26T21:28:43.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:42 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:43.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:42 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:43.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:42 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:28:45.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:44 smithi142 ceph-mon[32944]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:28:45.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:44 smithi046 ceph-mon[29201]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:28:45.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:44 smithi050 ceph-mon[32753]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:28:45.447 INFO:teuthology.orchestra.run.smithi050.stdout:Created osd(s) 4 on host 'smithi050' 2024-04-26T21:28:45.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:45.446+0000 7fe0667fc700 1 -- 172.21.15.50:0/837651623 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe07406b810 con 0x7fe0600a3c40 2024-04-26T21:28:45.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:45.448+0000 7fe07b534700 1 -- 172.21.15.50:0/837651623 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe0600a3c40 msgr2=0x7fe0600a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:45.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:45.448+0000 7fe07b534700 1 --2- 172.21.15.50:0/837651623 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe0600a3c40 0x7fe0600a6100 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe068000c00 tx=0x7fe068005fb0 comp rx=0 tx=0).stop 2024-04-26T21:28:45.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:45.449+0000 7fe07b534700 1 -- 172.21.15.50:0/837651623 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe074103f20 msgr2=0x7fe0741a0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:45.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:45.449+0000 7fe07b534700 1 --2- 172.21.15.50:0/837651623 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe074103f20 0x7fe0741a0c00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe06c00c370 tx=0x7fe06c00c680 comp rx=0 tx=0).stop 2024-04-26T21:28:45.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:45.449+0000 7fe07b534700 1 -- 172.21.15.50:0/837651623 shutdown_connections 2024-04-26T21:28:45.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:45.449+0000 7fe07b534700 1 --2- 172.21.15.50:0/837651623 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe074103f20 0x7fe0741a0c00 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:45.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:45.449+0000 7fe07b534700 1 --2- 172.21.15.50:0/837651623 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe074102380 0x7fe074199690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:45.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:45.449+0000 7fe07b534700 1 --2- 172.21.15.50:0/837651623 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe0600a3c40 0x7fe0600a6100 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:45.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:45.449+0000 7fe07b534700 1 --2- 172.21.15.50:0/837651623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe074103580 0x7fe074199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:45.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:45.449+0000 7fe07b534700 1 -- 172.21.15.50:0/837651623 >> 172.21.15.50:0/837651623 conn(0x7fe0740fd920 msgr2=0x7fe07406b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:45.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:45.449+0000 7fe07b534700 1 -- 172.21.15.50:0/837651623 shutdown_connections 2024-04-26T21:28:45.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:28:45.449+0000 7fe07b534700 1 -- 172.21.15.50:0/837651623 wait complete. 2024-04-26T21:28:46.079 DEBUG:teuthology.orchestra.run.smithi050:osd.4> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.4.service 2024-04-26T21:28:46.081 INFO:tasks.cephadm:Deploying osd.5 on smithi142 with /dev/vg_nvme/lv_3... 2024-04-26T21:28:46.081 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T21:28:46.328 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:28:46.680 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:46 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:46.680 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:46 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:46.680 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:46 smithi142 ceph-mon[32944]: from='osd.4 [v2:172.21.15.50:6808/3177333250,v1:172.21.15.50:6809/3177333250]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T21:28:46.680 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:46 smithi142 ceph-mon[32944]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T21:28:46.680 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:46 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:46.680 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:46 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:46.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:46 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:46.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:46 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:46.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:46 smithi046 ceph-mon[29201]: from='osd.4 [v2:172.21.15.50:6808/3177333250,v1:172.21.15.50:6809/3177333250]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T21:28:46.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:46 smithi046 ceph-mon[29201]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T21:28:46.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:46 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:46.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:46 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:46.854 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:46 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:46.854 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:46 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:46.855 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:46 smithi050 ceph-mon[32753]: from='osd.4 [v2:172.21.15.50:6808/3177333250,v1:172.21.15.50:6809/3177333250]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T21:28:46.855 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:46 smithi050 ceph-mon[32753]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T21:28:46.855 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:46 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:46.855 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:46 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:47.692 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:47 smithi142 ceph-mon[32944]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:28:47.693 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:47 smithi142 ceph-mon[32944]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T21:28:47.693 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:47 smithi142 ceph-mon[32944]: from='osd.4 [v2:172.21.15.50:6808/3177333250,v1:172.21.15.50:6809/3177333250]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T21:28:47.693 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:47 smithi142 ceph-mon[32944]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T21:28:47.693 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:47 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:28:47.693 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:47 smithi142 ceph-mon[32944]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T21:28:47.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:47 smithi046 ceph-mon[29201]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:28:47.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:47 smithi046 ceph-mon[29201]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T21:28:47.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:47 smithi046 ceph-mon[29201]: from='osd.4 [v2:172.21.15.50:6808/3177333250,v1:172.21.15.50:6809/3177333250]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T21:28:47.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:47 smithi046 ceph-mon[29201]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T21:28:47.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:47 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:28:47.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:47 smithi046 ceph-mon[29201]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T21:28:47.849 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:47 smithi050 ceph-mon[32753]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:28:47.849 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:47 smithi050 ceph-mon[32753]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T21:28:47.849 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:47 smithi050 ceph-mon[32753]: from='osd.4 [v2:172.21.15.50:6808/3177333250,v1:172.21.15.50:6809/3177333250]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T21:28:47.849 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:47 smithi050 ceph-mon[32753]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T21:28:47.849 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:47 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:28:47.849 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:47 smithi050 ceph-mon[32753]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T21:28:48.118 INFO:journalctl@ceph.osd.4.smithi050.stdout:Apr 26 21:28:47 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-4[44295]: 2024-04-26T21:28:47.900+0000 7f2f93e02700 -1 osd.4 0 waiting for initial osdmap 2024-04-26T21:28:48.118 INFO:journalctl@ceph.osd.4.smithi050.stdout:Apr 26 21:28:47 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-4[44295]: 2024-04-26T21:28:47.906+0000 7f2f8bbed700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T21:28:48.369 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T21:28:48.369 INFO:teuthology.orchestra.run.smithi142.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T21:28:48.369 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10+0 records in 2024-04-26T21:28:48.369 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records out 2024-04-26T21:28:48.369 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011194 s, 937 MB/s 2024-04-26T21:28:48.369 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping successful for: 2024-04-26T21:28:48.664 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:48 smithi142 ceph-mon[32944]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T21:28:48.664 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:48 smithi142 ceph-mon[32944]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T21:28:48.665 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:48 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:28:48.665 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:48 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:28:48.665 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:48 smithi142 ceph-mon[32944]: from='osd.4 ' entity='osd.4' 2024-04-26T21:28:48.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:48 smithi046 ceph-mon[29201]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T21:28:48.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:48 smithi046 ceph-mon[29201]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T21:28:48.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:48 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:28:48.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:48 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:28:48.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:48 smithi046 ceph-mon[29201]: from='osd.4 ' entity='osd.4' 2024-04-26T21:28:48.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:48 smithi050 ceph-mon[32753]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T21:28:48.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:48 smithi050 ceph-mon[32753]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T21:28:48.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:48 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:28:48.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:48 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:28:48.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:48 smithi050 ceph-mon[32753]: from='osd.4 ' entity='osd.4' 2024-04-26T21:28:49.070 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi142:vg_nvme/lv_3 2024-04-26T21:28:49.310 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:28:49.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:49 smithi050 ceph-mon[32753]: purged_snaps scrub starts 2024-04-26T21:28:49.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:49 smithi050 ceph-mon[32753]: purged_snaps scrub ok 2024-04-26T21:28:49.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:49 smithi050 ceph-mon[32753]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:28:49.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:49 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:28:49.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:49 smithi050 ceph-mon[32753]: osd.4 [v2:172.21.15.50:6808/3177333250,v1:172.21.15.50:6809/3177333250] boot 2024-04-26T21:28:49.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:49 smithi050 ceph-mon[32753]: osdmap e33: 5 total, 5 up, 5 in 2024-04-26T21:28:49.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:49 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:28:49.719 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:49 smithi142 ceph-mon[32944]: purged_snaps scrub starts 2024-04-26T21:28:49.719 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:49 smithi142 ceph-mon[32944]: purged_snaps scrub ok 2024-04-26T21:28:49.719 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:49 smithi142 ceph-mon[32944]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:28:49.719 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:49 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:28:49.719 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:49 smithi142 ceph-mon[32944]: osd.4 [v2:172.21.15.50:6808/3177333250,v1:172.21.15.50:6809/3177333250] boot 2024-04-26T21:28:49.719 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:49 smithi142 ceph-mon[32944]: osdmap e33: 5 total, 5 up, 5 in 2024-04-26T21:28:49.719 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:49 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:28:49.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:49 smithi046 ceph-mon[29201]: purged_snaps scrub starts 2024-04-26T21:28:49.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:49 smithi046 ceph-mon[29201]: purged_snaps scrub ok 2024-04-26T21:28:49.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:49 smithi046 ceph-mon[29201]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:28:49.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:49 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:28:49.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:49 smithi046 ceph-mon[29201]: osd.4 [v2:172.21.15.50:6808/3177333250,v1:172.21.15.50:6809/3177333250] boot 2024-04-26T21:28:49.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:49 smithi046 ceph-mon[29201]: osdmap e33: 5 total, 5 up, 5 in 2024-04-26T21:28:49.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:49 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:28:50.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.986+0000 7fd7502b2700 1 -- 172.21.15.142:0/1515365884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd748103470 msgr2=0x7fd748103890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:50.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.986+0000 7fd7502b2700 1 --2- 172.21.15.142:0/1515365884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd748103470 0x7fd748103890 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd744009990 tx=0x7fd744009ca0 comp rx=0 tx=0).stop 2024-04-26T21:28:50.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.986+0000 7fd7502b2700 1 -- 172.21.15.142:0/1515365884 shutdown_connections 2024-04-26T21:28:50.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.986+0000 7fd7502b2700 1 --2- 172.21.15.142:0/1515365884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd748105010 0x7fd74810b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:50.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.986+0000 7fd7502b2700 1 --2- 172.21.15.142:0/1515365884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd748103470 0x7fd748103890 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:50.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.986+0000 7fd7502b2700 1 --2- 172.21.15.142:0/1515365884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd748104670 0x7fd748104ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:50.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.986+0000 7fd7502b2700 1 -- 172.21.15.142:0/1515365884 >> 172.21.15.142:0/1515365884 conn(0x7fd7480fea10 msgr2=0x7fd748100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:50.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.986+0000 7fd7502b2700 1 -- 172.21.15.142:0/1515365884 shutdown_connections 2024-04-26T21:28:50.990 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.986+0000 7fd7502b2700 1 -- 172.21.15.142:0/1515365884 wait complete. 2024-04-26T21:28:50.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.986+0000 7fd7502b2700 1 Processor -- start 2024-04-26T21:28:50.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.986+0000 7fd7502b2700 1 -- start start 2024-04-26T21:28:50.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.986+0000 7fd7502b2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd748103470 0x7fd748199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:50.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.986+0000 7fd7502b2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd748104670 0x7fd748199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:50.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.986+0000 7fd7502b2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd748105010 0x7fd7481a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:50.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.986+0000 7fd7502b2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd74810ef60 con 0x7fd748104670 2024-04-26T21:28:50.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.986+0000 7fd7502b2700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd74810ede0 con 0x7fd748103470 2024-04-26T21:28:50.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.986+0000 7fd7502b2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd74810f0e0 con 0x7fd748105010 2024-04-26T21:28:50.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.987+0000 7fd74e84f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd748105010 0x7fd7481a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:50.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.987+0000 7fd74e84f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd748105010 0x7fd7481a0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:43712/0 (socket says 172.21.15.142:43712) 2024-04-26T21:28:50.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.987+0000 7fd74e84f700 1 -- 172.21.15.142:0/3198783019 learned_addr learned my addr 172.21.15.142:0/3198783019 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-04-26T21:28:50.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.987+0000 7fd74d84d700 1 --2- 172.21.15.142:0/3198783019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd748104670 0x7fd748199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:50.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.987+0000 7fd74e04e700 1 --2- 172.21.15.142:0/3198783019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd748103470 0x7fd748199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:50.991 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.987+0000 7fd74e84f700 1 -- 172.21.15.142:0/3198783019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd748103470 msgr2=0x7fd748199690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:50.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.987+0000 7fd74e84f700 1 --2- 172.21.15.142:0/3198783019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd748103470 0x7fd748199690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:50.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.987+0000 7fd74e84f700 1 -- 172.21.15.142:0/3198783019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd748104670 msgr2=0x7fd748199bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:50.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.987+0000 7fd74e84f700 1 --2- 172.21.15.142:0/3198783019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd748104670 0x7fd748199bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:50.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.987+0000 7fd74e84f700 1 -- 172.21.15.142:0/3198783019 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7481a12a0 con 0x7fd748105010 2024-04-26T21:28:50.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.987+0000 7fd74e04e700 1 --2- 172.21.15.142:0/3198783019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd748103470 0x7fd748199690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:28:50.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.987+0000 7fd74e84f700 1 --2- 172.21.15.142:0/3198783019 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd748105010 0x7fd7481a0c00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd740007f10 tx=0x7fd74000d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:50.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.987+0000 7fd73f7fe700 1 -- 172.21.15.142:0/3198783019 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd740004bb0 con 0x7fd748105010 2024-04-26T21:28:50.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.987+0000 7fd7502b2700 1 -- 172.21.15.142:0/3198783019 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7481a1590 con 0x7fd748105010 2024-04-26T21:28:50.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.987+0000 7fd73f7fe700 1 -- 172.21.15.142:0/3198783019 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd740004d50 con 0x7fd748105010 2024-04-26T21:28:50.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.987+0000 7fd73f7fe700 1 -- 172.21.15.142:0/3198783019 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd740005a30 con 0x7fd748105010 2024-04-26T21:28:50.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.987+0000 7fd7502b2700 1 -- 172.21.15.142:0/3198783019 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7481a1b20 con 0x7fd748105010 2024-04-26T21:28:50.992 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.988+0000 7fd7502b2700 1 -- 172.21.15.142:0/3198783019 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd74806a110 con 0x7fd748105010 2024-04-26T21:28:50.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.989+0000 7fd73f7fe700 1 -- 172.21.15.142:0/3198783019 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd74001a020 con 0x7fd748105010 2024-04-26T21:28:50.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.990+0000 7fd73f7fe700 1 --2- 172.21.15.142:0/3198783019 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fd7340a3c40 0x7fd7340a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:50.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.990+0000 7fd73f7fe700 1 -- 172.21.15.142:0/3198783019 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7fd7400cf120 con 0x7fd748105010 2024-04-26T21:28:50.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.990+0000 7fd74e04e700 1 --2- 172.21.15.142:0/3198783019 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fd7340a3c40 0x7fd7340a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:50.993 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.991+0000 7fd74e04e700 1 --2- 172.21.15.142:0/3198783019 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fd7340a3c40 0x7fd7340a6100 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd7440038a0 tx=0x7fd744005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:50.995 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:50.994+0000 7fd73f7fe700 1 -- 172.21.15.142:0/3198783019 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd740010070 con 0x7fd748105010 2024-04-26T21:28:51.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:50 smithi142 ceph-mon[32944]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:28:51.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:50 smithi142 ceph-mon[32944]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T21:28:51.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:50 smithi142 ceph-mon[32944]: Detected new or changed devices on smithi050 2024-04-26T21:28:51.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:50 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:51.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:50 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:51.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:50 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:28:51.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:50 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:28:51.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:50 smithi142 ceph-mon[32944]: Adjusting osd_memory_target on smithi050 to 8546M 2024-04-26T21:28:51.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:50 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:51.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:50 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:51.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:50 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:28:51.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:50 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:51.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:50 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:28:51.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:50 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:51.043 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:50 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:28:51.043 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:50 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:51.203 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:28:51.201+0000 7fd7502b2700 1 -- 172.21.15.142:0/3198783019 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fd748063860 con 0x7fd7340a3c40 2024-04-26T21:28:51.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:50 smithi046 ceph-mon[29201]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:28:51.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:50 smithi046 ceph-mon[29201]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T21:28:51.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:50 smithi046 ceph-mon[29201]: Detected new or changed devices on smithi050 2024-04-26T21:28:51.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:50 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:51.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:50 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:51.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:50 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:28:51.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:50 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:28:51.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:50 smithi046 ceph-mon[29201]: Adjusting osd_memory_target on smithi050 to 8546M 2024-04-26T21:28:51.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:50 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:51.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:50 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:51.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:50 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:28:51.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:50 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:51.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:50 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:28:51.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:50 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:51.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:50 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:28:51.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:50 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:51.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:50 smithi050 ceph-mon[32753]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:28:51.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:50 smithi050 ceph-mon[32753]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T21:28:51.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:50 smithi050 ceph-mon[32753]: Detected new or changed devices on smithi050 2024-04-26T21:28:51.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:50 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:51.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:50 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:51.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:50 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:28:51.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:50 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:28:51.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:50 smithi050 ceph-mon[32753]: Adjusting osd_memory_target on smithi050 to 8546M 2024-04-26T21:28:51.370 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:50 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:51.370 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:50 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:51.370 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:50 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:28:51.370 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:50 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:51.370 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:50 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:28:51.370 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:50 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:51.370 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:50 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:28:51.370 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:50 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:28:52.119 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:51 smithi142 ceph-mon[32944]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:28:52.120 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:51 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:28:52.120 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:51 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:28:52.120 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:51 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:52.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:51 smithi046 ceph-mon[29201]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:28:52.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:51 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:28:52.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:51 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:28:52.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:51 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:52.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:51 smithi050 ceph-mon[32753]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:28:52.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:51 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:28:52.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:51 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:28:52.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:51 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:52.974 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:52 smithi142 ceph-mon[32944]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:28:53.206 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:52 smithi046 ceph-mon[29201]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:28:53.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:52 smithi050 ceph-mon[32753]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:28:54.185 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:53 smithi142 ceph-mon[32944]: from='client.? 172.21.15.142:0/2957116961' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7d65851-ebe1-45e2-833d-8022446f2698"}]: dispatch 2024-04-26T21:28:54.186 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:53 smithi142 ceph-mon[32944]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7d65851-ebe1-45e2-833d-8022446f2698"}]: dispatch 2024-04-26T21:28:54.186 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:53 smithi142 ceph-mon[32944]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7d65851-ebe1-45e2-833d-8022446f2698"}]': finished 2024-04-26T21:28:54.186 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:53 smithi142 ceph-mon[32944]: osdmap e35: 6 total, 5 up, 6 in 2024-04-26T21:28:54.186 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:53 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:28:54.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:53 smithi046 ceph-mon[29201]: from='client.? 172.21.15.142:0/2957116961' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7d65851-ebe1-45e2-833d-8022446f2698"}]: dispatch 2024-04-26T21:28:54.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:53 smithi046 ceph-mon[29201]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7d65851-ebe1-45e2-833d-8022446f2698"}]: dispatch 2024-04-26T21:28:54.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:53 smithi046 ceph-mon[29201]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7d65851-ebe1-45e2-833d-8022446f2698"}]': finished 2024-04-26T21:28:54.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:53 smithi046 ceph-mon[29201]: osdmap e35: 6 total, 5 up, 6 in 2024-04-26T21:28:54.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:53 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:28:54.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:53 smithi050 ceph-mon[32753]: from='client.? 172.21.15.142:0/2957116961' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7d65851-ebe1-45e2-833d-8022446f2698"}]: dispatch 2024-04-26T21:28:54.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:53 smithi050 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7d65851-ebe1-45e2-833d-8022446f2698"}]: dispatch 2024-04-26T21:28:54.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:53 smithi050 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7d65851-ebe1-45e2-833d-8022446f2698"}]': finished 2024-04-26T21:28:54.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:53 smithi050 ceph-mon[32753]: osdmap e35: 6 total, 5 up, 6 in 2024-04-26T21:28:54.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:53 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:28:55.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:54 smithi142 ceph-mon[32944]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:28:55.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:54 smithi142 ceph-mon[32944]: from='client.? 172.21.15.142:0/1651229818' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:28:55.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:54 smithi046 ceph-mon[29201]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:28:55.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:54 smithi046 ceph-mon[29201]: from='client.? 172.21.15.142:0/1651229818' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:28:55.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:54 smithi050 ceph-mon[32753]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:28:55.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:54 smithi050 ceph-mon[32753]: from='client.? 172.21.15.142:0/1651229818' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:28:57.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:56 smithi142 ceph-mon[32944]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:28:57.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:56 smithi046 ceph-mon[29201]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:28:57.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:56 smithi050 ceph-mon[32753]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:28:59.090 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:28:58 smithi142 ceph-mon[32944]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:28:59.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:28:58 smithi046 ceph-mon[29201]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:28:59.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:28:58 smithi050 ceph-mon[32753]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:29:01.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:00 smithi142 ceph-mon[32944]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:29:01.306 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:00 smithi046 ceph-mon[29201]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:29:01.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:00 smithi050 ceph-mon[32753]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:29:02.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:01 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T21:29:02.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:01 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:02.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:01 smithi142 ceph-mon[32944]: Deploying daemon osd.5 on smithi142 2024-04-26T21:29:02.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:01 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T21:29:02.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:01 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:02.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:01 smithi046 ceph-mon[29201]: Deploying daemon osd.5 on smithi142 2024-04-26T21:29:02.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:01 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T21:29:02.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:01 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:02.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:01 smithi050 ceph-mon[32753]: Deploying daemon osd.5 on smithi142 2024-04-26T21:29:03.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:02 smithi142 ceph-mon[32944]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:29:03.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:02 smithi046 ceph-mon[29201]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:29:03.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:02 smithi050 ceph-mon[32753]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:29:05.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:04 smithi142 ceph-mon[32944]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:29:05.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:04 smithi046 ceph-mon[29201]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:29:05.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:04 smithi050 ceph-mon[32753]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:29:07.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:07 smithi142 ceph-mon[32944]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:29:07.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:07 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:07.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:07 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:07.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:07 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:29:07.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:07 smithi046 ceph-mon[29201]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:29:07.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:07 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:07.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:07 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:07.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:07 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:29:07.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:07 smithi050 ceph-mon[32753]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:29:07.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:07 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:07.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:07 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:07.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:07 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:29:09.760 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:09 smithi142 ceph-mon[32944]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:29:09.760 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:09 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:09.760 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:09 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:09.760 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:09 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:09.760 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:09 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:29:09.760 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:09 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:09.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:09 smithi046 ceph-mon[29201]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:29:09.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:09 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:09.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:09 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:09.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:09 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:09.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:09 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:29:09.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:09 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:09.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:09 smithi050 ceph-mon[32753]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:29:09.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:09 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:09.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:09 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:09.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:09 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:09.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:09 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:29:09.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:09 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:10.123 INFO:teuthology.orchestra.run.smithi142.stdout:Created osd(s) 5 on host 'smithi142' 2024-04-26T21:29:10.123 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:29:10.122+0000 7fd73f7fe700 1 -- 172.21.15.142:0/3198783019 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd748063860 con 0x7fd7340a3c40 2024-04-26T21:29:10.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:29:10.124+0000 7fd7502b2700 1 -- 172.21.15.142:0/3198783019 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fd7340a3c40 msgr2=0x7fd7340a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:10.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:29:10.124+0000 7fd7502b2700 1 --2- 172.21.15.142:0/3198783019 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fd7340a3c40 0x7fd7340a6100 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd7440038a0 tx=0x7fd744005fd0 comp rx=0 tx=0).stop 2024-04-26T21:29:10.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:29:10.124+0000 7fd7502b2700 1 -- 172.21.15.142:0/3198783019 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd748105010 msgr2=0x7fd7481a0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:10.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:29:10.124+0000 7fd7502b2700 1 --2- 172.21.15.142:0/3198783019 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd748105010 0x7fd7481a0c00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd740007f10 tx=0x7fd74000d3b0 comp rx=0 tx=0).stop 2024-04-26T21:29:10.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:29:10.124+0000 7fd7502b2700 1 -- 172.21.15.142:0/3198783019 shutdown_connections 2024-04-26T21:29:10.125 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:29:10.124+0000 7fd7502b2700 1 --2- 172.21.15.142:0/3198783019 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd748105010 0x7fd7481a0c00 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:10.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:29:10.124+0000 7fd7502b2700 1 --2- 172.21.15.142:0/3198783019 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd748103470 0x7fd748199690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:10.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:29:10.124+0000 7fd7502b2700 1 --2- 172.21.15.142:0/3198783019 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fd7340a3c40 0x7fd7340a6100 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:10.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:29:10.124+0000 7fd7502b2700 1 --2- 172.21.15.142:0/3198783019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd748104670 0x7fd748199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:10.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:29:10.124+0000 7fd7502b2700 1 -- 172.21.15.142:0/3198783019 >> 172.21.15.142:0/3198783019 conn(0x7fd7480fea10 msgr2=0x7fd748107ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:10.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:29:10.124+0000 7fd7502b2700 1 -- 172.21.15.142:0/3198783019 shutdown_connections 2024-04-26T21:29:10.126 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:29:10.124+0000 7fd7502b2700 1 -- 172.21.15.142:0/3198783019 wait complete. 2024-04-26T21:29:10.747 DEBUG:teuthology.orchestra.run.smithi142:osd.5> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.5.service 2024-04-26T21:29:10.749 INFO:tasks.cephadm:Deploying osd.6 on smithi046 with /dev/vg_nvme/lv_2... 2024-04-26T21:29:10.750 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T21:29:11.006 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:29:11.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:11 smithi050 ceph-mon[32753]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:29:11.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:11 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:11.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:11 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:11.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:11 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:29:11.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:11 smithi050 ceph-mon[32753]: from='osd.5 [v2:172.21.15.142:6808/4113107737,v1:172.21.15.142:6809/4113107737]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T21:29:11.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:11 smithi050 ceph-mon[32753]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T21:29:11.379 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:11 smithi046 ceph-mon[29201]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:29:11.379 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:11 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:11.379 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:11 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:11.379 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:11 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:29:11.379 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:11 smithi046 ceph-mon[29201]: from='osd.5 [v2:172.21.15.142:6808/4113107737,v1:172.21.15.142:6809/4113107737]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T21:29:11.379 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:11 smithi046 ceph-mon[29201]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T21:29:11.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:11 smithi142 ceph-mon[32944]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:29:11.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:11 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:11.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:11 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:11.542 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:11 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:29:11.542 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:11 smithi142 ceph-mon[32944]: from='osd.5 [v2:172.21.15.142:6808/4113107737,v1:172.21.15.142:6809/4113107737]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T21:29:11.542 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:11 smithi142 ceph-mon[32944]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T21:29:12.191 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:12 smithi142 ceph-mon[32944]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T21:29:12.191 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:12 smithi142 ceph-mon[32944]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T21:29:12.191 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:12 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:29:12.191 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:12 smithi142 ceph-mon[32944]: from='osd.5 [v2:172.21.15.142:6808/4113107737,v1:172.21.15.142:6809/4113107737]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-04-26T21:29:12.191 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:12 smithi142 ceph-mon[32944]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-04-26T21:29:12.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:12 smithi050 ceph-mon[32753]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T21:29:12.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:12 smithi050 ceph-mon[32753]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T21:29:12.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:12 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:29:12.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:12 smithi050 ceph-mon[32753]: from='osd.5 [v2:172.21.15.142:6808/4113107737,v1:172.21.15.142:6809/4113107737]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-04-26T21:29:12.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:12 smithi050 ceph-mon[32753]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-04-26T21:29:12.463 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:12 smithi046 ceph-mon[29201]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T21:29:12.463 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:12 smithi046 ceph-mon[29201]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T21:29:12.463 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:12 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:29:12.463 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:12 smithi046 ceph-mon[29201]: from='osd.5 [v2:172.21.15.142:6808/4113107737,v1:172.21.15.142:6809/4113107737]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-04-26T21:29:12.463 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:12 smithi046 ceph-mon[29201]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-04-26T21:29:12.735 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:29:12 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:29:12.559+0000 7f013d015700 -1 osd.5 0 waiting for initial osdmap 2024-04-26T21:29:12.735 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:29:12 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:29:12.565+0000 7f0134e00700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T21:29:13.209 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T21:29:13.209 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T21:29:13.209 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2024-04-26T21:29:13.209 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2024-04-26T21:29:13.209 INFO:teuthology.orchestra.run.smithi046.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00838133 s, 1.3 GB/s 2024-04-26T21:29:13.209 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2024-04-26T21:29:13.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:13 smithi050 ceph-mon[32753]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:29:13.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:13 smithi050 ceph-mon[32753]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-04-26T21:29:13.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:13 smithi050 ceph-mon[32753]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T21:29:13.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:13 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:29:13.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:13 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:29:13.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:13 smithi050 ceph-mon[32753]: from='osd.5 ' entity='osd.5' 2024-04-26T21:29:13.505 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:13 smithi142 ceph-mon[32944]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:29:13.505 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:13 smithi142 ceph-mon[32944]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-04-26T21:29:13.505 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:13 smithi142 ceph-mon[32944]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T21:29:13.505 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:13 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:29:13.505 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:13 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:29:13.505 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:13 smithi142 ceph-mon[32944]: from='osd.5 ' entity='osd.5' 2024-04-26T21:29:13.523 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:13 smithi046 ceph-mon[29201]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:29:13.523 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:13 smithi046 ceph-mon[29201]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-04-26T21:29:13.523 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:13 smithi046 ceph-mon[29201]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T21:29:13.523 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:13 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:29:13.523 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:13 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:29:13.524 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:13 smithi046 ceph-mon[29201]: from='osd.5 ' entity='osd.5' 2024-04-26T21:29:13.915 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi046:vg_nvme/lv_2 2024-04-26T21:29:14.181 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:29:14.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:14 smithi046 ceph-mon[29201]: purged_snaps scrub starts 2024-04-26T21:29:14.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:14 smithi046 ceph-mon[29201]: purged_snaps scrub ok 2024-04-26T21:29:14.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:14 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:29:14.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:14 smithi046 ceph-mon[29201]: osd.5 [v2:172.21.15.142:6808/4113107737,v1:172.21.15.142:6809/4113107737] boot 2024-04-26T21:29:14.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:14 smithi046 ceph-mon[29201]: osdmap e38: 6 total, 6 up, 6 in 2024-04-26T21:29:14.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:14 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:29:14.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:14 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:14.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:14 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:14.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:14 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:29:14.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:14 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:29:14.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:14 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:14.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:14 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:14.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:14 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:29:14.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:14 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:14.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:14 smithi050 ceph-mon[32753]: purged_snaps scrub starts 2024-04-26T21:29:14.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:14 smithi050 ceph-mon[32753]: purged_snaps scrub ok 2024-04-26T21:29:14.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:14 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:29:14.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:14 smithi050 ceph-mon[32753]: osd.5 [v2:172.21.15.142:6808/4113107737,v1:172.21.15.142:6809/4113107737] boot 2024-04-26T21:29:14.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:14 smithi050 ceph-mon[32753]: osdmap e38: 6 total, 6 up, 6 in 2024-04-26T21:29:14.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:14 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:29:14.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:14 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:14.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:14 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:14.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:14 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:29:14.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:14 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:29:14.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:14 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:14.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:14 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:14.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:14 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:29:14.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:14 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:14.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:14 smithi142 ceph-mon[32944]: purged_snaps scrub starts 2024-04-26T21:29:14.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:14 smithi142 ceph-mon[32944]: purged_snaps scrub ok 2024-04-26T21:29:14.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:14 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:29:14.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:14 smithi142 ceph-mon[32944]: osd.5 [v2:172.21.15.142:6808/4113107737,v1:172.21.15.142:6809/4113107737] boot 2024-04-26T21:29:14.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:14 smithi142 ceph-mon[32944]: osdmap e38: 6 total, 6 up, 6 in 2024-04-26T21:29:14.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:14 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:29:14.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:14 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:14.542 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:14 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:14.542 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:14 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:29:14.542 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:14 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:29:14.542 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:14 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:14.542 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:14 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:14.542 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:14 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:29:14.542 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:14 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:15.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:15 smithi050 ceph-mon[32753]: Detected new or changed devices on smithi142 2024-04-26T21:29:15.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:15 smithi050 ceph-mon[32753]: Adjusting osd_memory_target on smithi142 to 8544M 2024-04-26T21:29:15.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:15 smithi050 ceph-mon[32753]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:29:15.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:15 smithi050 ceph-mon[32753]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T21:29:15.472 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:15 smithi046 ceph-mon[29201]: Detected new or changed devices on smithi142 2024-04-26T21:29:15.473 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:15 smithi046 ceph-mon[29201]: Adjusting osd_memory_target on smithi142 to 8544M 2024-04-26T21:29:15.473 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:15 smithi046 ceph-mon[29201]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:29:15.473 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:15 smithi046 ceph-mon[29201]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T21:29:15.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:15 smithi142 ceph-mon[32944]: Detected new or changed devices on smithi142 2024-04-26T21:29:15.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:15 smithi142 ceph-mon[32944]: Adjusting osd_memory_target on smithi142 to 8544M 2024-04-26T21:29:15.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:15 smithi142 ceph-mon[32944]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 158 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:29:15.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:15 smithi142 ceph-mon[32944]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T21:29:16.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.047+0000 7f11aec9b700 1 -- 172.21.15.46:0/2212149379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11a8103f20 msgr2=0x7f11a810a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:16.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.047+0000 7f11aec9b700 1 --2- 172.21.15.46:0/2212149379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11a8103f20 0x7f11a810a7d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f11a000b0d0 tx=0x7f11a000b3e0 comp rx=0 tx=0).stop 2024-04-26T21:29:16.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.047+0000 7f11aec9b700 1 -- 172.21.15.46:0/2212149379 shutdown_connections 2024-04-26T21:29:16.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.047+0000 7f11aec9b700 1 --2- 172.21.15.46:0/2212149379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11a8103f20 0x7f11a810a7d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:16.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.047+0000 7f11aec9b700 1 --2- 172.21.15.46:0/2212149379 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11a8102380 0x7f11a81027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:16.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.047+0000 7f11aec9b700 1 --2- 172.21.15.46:0/2212149379 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11a8103580 0x7f11a81039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:16.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.047+0000 7f11aec9b700 1 -- 172.21.15.46:0/2212149379 >> 172.21.15.46:0/2212149379 conn(0x7f11a80fd920 msgr2=0x7f11a80ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:16.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.047+0000 7f11aec9b700 1 -- 172.21.15.46:0/2212149379 shutdown_connections 2024-04-26T21:29:16.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.047+0000 7f11aec9b700 1 -- 172.21.15.46:0/2212149379 wait complete. 2024-04-26T21:29:16.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.048+0000 7f11aec9b700 1 Processor -- start 2024-04-26T21:29:16.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.048+0000 7f11aec9b700 1 -- start start 2024-04-26T21:29:16.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.048+0000 7f11aec9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11a8102380 0x7f11a806bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:16.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.048+0000 7f11aec9b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11a8103580 0x7f11a819e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:16.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.048+0000 7f11aec9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11a8103f20 0x7f11a81a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:16.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.048+0000 7f11aec9b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f11a810de20 con 0x7f11a8103f20 2024-04-26T21:29:16.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.048+0000 7f11aec9b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f11a810dca0 con 0x7f11a8102380 2024-04-26T21:29:16.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.048+0000 7f11aec9b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f11a810dfa0 con 0x7f11a8103580 2024-04-26T21:29:16.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.048+0000 7f11ad238700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11a8103f20 0x7f11a81a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:16.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.048+0000 7f11ad238700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11a8103f20 0x7f11a81a0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36958/0 (socket says 172.21.15.46:36958) 2024-04-26T21:29:16.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.048+0000 7f11ad238700 1 -- 172.21.15.46:0/1706334670 learned_addr learned my addr 172.21.15.46:0/1706334670 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:29:16.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.048+0000 7f11aca37700 1 --2- 172.21.15.46:0/1706334670 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11a8102380 0x7f11a806bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:16.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.048+0000 7f11a7fff700 1 --2- 172.21.15.46:0/1706334670 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11a8103580 0x7f11a819e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:16.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.049+0000 7f11ad238700 1 -- 172.21.15.46:0/1706334670 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11a8103580 msgr2=0x7f11a819e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:16.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.049+0000 7f11ad238700 1 --2- 172.21.15.46:0/1706334670 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11a8103580 0x7f11a819e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:16.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.049+0000 7f11ad238700 1 -- 172.21.15.46:0/1706334670 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11a8102380 msgr2=0x7f11a806bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:16.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.049+0000 7f11ad238700 1 --2- 172.21.15.46:0/1706334670 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11a8102380 0x7f11a806bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:16.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.049+0000 7f11ad238700 1 -- 172.21.15.46:0/1706334670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11a81a1200 con 0x7f11a8103f20 2024-04-26T21:29:16.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.049+0000 7f11ad238700 1 --2- 172.21.15.46:0/1706334670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11a8103f20 0x7f11a81a0930 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f11a000d010 tx=0x7f11a0007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:16.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.049+0000 7f11a5ffb700 1 -- 172.21.15.46:0/1706334670 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11a0021070 con 0x7f11a8103f20 2024-04-26T21:29:16.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.049+0000 7f11a5ffb700 1 -- 172.21.15.46:0/1706334670 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11a0007e50 con 0x7f11a8103f20 2024-04-26T21:29:16.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.049+0000 7f11aec9b700 1 -- 172.21.15.46:0/1706334670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11a81a1490 con 0x7f11a8103f20 2024-04-26T21:29:16.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.049+0000 7f11a5ffb700 1 -- 172.21.15.46:0/1706334670 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11a001bb00 con 0x7f11a8103f20 2024-04-26T21:29:16.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.049+0000 7f11aec9b700 1 -- 172.21.15.46:0/1706334670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11a81a1940 con 0x7f11a8103f20 2024-04-26T21:29:16.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.050+0000 7f11aec9b700 1 -- 172.21.15.46:0/1706334670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11a806a110 con 0x7f11a8103f20 2024-04-26T21:29:16.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.052+0000 7f11a5ffb700 1 -- 172.21.15.46:0/1706334670 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f11a0024070 con 0x7f11a8103f20 2024-04-26T21:29:16.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.052+0000 7f11a5ffb700 1 --2- 172.21.15.46:0/1706334670 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f11980a3c40 0x7f11980a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:16.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.052+0000 7f11aca37700 1 --2- 172.21.15.46:0/1706334670 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f11980a3c40 0x7f11980a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:16.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.052+0000 7f11a5ffb700 1 -- 172.21.15.46:0/1706334670 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4562+0+0 (secure 0 0 0) 0x7f11a000f020 con 0x7f11a8103f20 2024-04-26T21:29:16.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.053+0000 7f11aca37700 1 --2- 172.21.15.46:0/1706334670 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f11980a3c40 0x7f11980a6100 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f11a81033e0 tx=0x7f119c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:16.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.056+0000 7f11a5ffb700 1 -- 172.21.15.46:0/1706334670 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f11a002b020 con 0x7f11a8103f20 2024-04-26T21:29:16.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:16.276+0000 7f11aec9b700 1 -- 172.21.15.46:0/1706334670 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f11a8063860 con 0x7f11980a3c40 2024-04-26T21:29:16.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:16 smithi050 ceph-mon[32753]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T21:29:16.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:16 smithi142 ceph-mon[32944]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T21:29:16.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:16 smithi046 ceph-mon[29201]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T21:29:17.212 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:17 smithi046 ceph-mon[29201]: pgmap v143: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T21:29:17.212 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:17 smithi046 ceph-mon[29201]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T21:29:17.212 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:17 smithi046 ceph-mon[29201]: from='client.14376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:29:17.212 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:17 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:29:17.212 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:17 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:29:17.212 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:17 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:17.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:17 smithi050 ceph-mon[32753]: pgmap v143: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T21:29:17.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:17 smithi050 ceph-mon[32753]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T21:29:17.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:17 smithi050 ceph-mon[32753]: from='client.14376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:29:17.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:17 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:29:17.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:17 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:29:17.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:17 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:17.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:17 smithi142 ceph-mon[32944]: pgmap v143: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T21:29:17.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:17 smithi142 ceph-mon[32944]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T21:29:17.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:17 smithi142 ceph-mon[32944]: from='client.14376 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:29:17.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:17 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:29:17.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:17 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:29:17.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:17 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:18.216 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:18 smithi046 ceph-mon[29201]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T21:29:18.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:18 smithi050 ceph-mon[32753]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T21:29:18.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:18 smithi142 ceph-mon[32944]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T21:29:19.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:19 smithi050 ceph-mon[32753]: pgmap v146: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T21:29:19.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:19 smithi142 ceph-mon[32944]: pgmap v146: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T21:29:19.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:19 smithi046 ceph-mon[29201]: pgmap v146: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T21:29:20.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:20 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/841218704' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2a3e96d7-d5bf-44ff-9134-29f1420c0e4c"}]: dispatch 2024-04-26T21:29:20.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:20 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/841218704' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2a3e96d7-d5bf-44ff-9134-29f1420c0e4c"}]': finished 2024-04-26T21:29:20.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:20 smithi142 ceph-mon[32944]: osdmap e43: 7 total, 6 up, 7 in 2024-04-26T21:29:20.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:20 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:29:20.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:20 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/183861642' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:29:20.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:20 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/841218704' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2a3e96d7-d5bf-44ff-9134-29f1420c0e4c"}]: dispatch 2024-04-26T21:29:20.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:20 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/841218704' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2a3e96d7-d5bf-44ff-9134-29f1420c0e4c"}]': finished 2024-04-26T21:29:20.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:20 smithi046 ceph-mon[29201]: osdmap e43: 7 total, 6 up, 7 in 2024-04-26T21:29:20.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:20 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:29:20.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:20 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/183861642' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:29:20.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:20 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/841218704' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2a3e96d7-d5bf-44ff-9134-29f1420c0e4c"}]: dispatch 2024-04-26T21:29:20.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:20 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/841218704' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2a3e96d7-d5bf-44ff-9134-29f1420c0e4c"}]': finished 2024-04-26T21:29:20.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:20 smithi050 ceph-mon[32753]: osdmap e43: 7 total, 6 up, 7 in 2024-04-26T21:29:20.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:20 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:29:20.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:20 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/183861642' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:29:21.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:21 smithi142 ceph-mon[32944]: pgmap v148: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-04-26T21:29:21.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:21 smithi046 ceph-mon[29201]: pgmap v148: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-04-26T21:29:21.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:21 smithi050 ceph-mon[32753]: pgmap v148: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-04-26T21:29:23.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:23 smithi142 ceph-mon[32944]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:23.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:23 smithi050 ceph-mon[32753]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:23.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:23 smithi046 ceph-mon[29201]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:25.449 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:25 smithi046 ceph-mon[29201]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:25.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:25 smithi142 ceph-mon[32944]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:25.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:25 smithi050 ceph-mon[32753]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:27.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:27 smithi142 ceph-mon[32944]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:27.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:27 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T21:29:27.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:27 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:27.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:27 smithi046 ceph-mon[29201]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:27.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:27 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T21:29:27.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:27 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:27.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:27 smithi050 ceph-mon[32753]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:27.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:27 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T21:29:27.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:27 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:28.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:28 smithi142 ceph-mon[32944]: Deploying daemon osd.6 on smithi046 2024-04-26T21:29:28.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:28 smithi046 ceph-mon[29201]: Deploying daemon osd.6 on smithi046 2024-04-26T21:29:28.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:28 smithi050 ceph-mon[32753]: Deploying daemon osd.6 on smithi046 2024-04-26T21:29:29.264 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:29 smithi046 ceph-mon[29201]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:29.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:29 smithi142 ceph-mon[32944]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:29.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:29 smithi050 ceph-mon[32753]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:31.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:31 smithi046 ceph-mon[29201]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:31.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:31 smithi142 ceph-mon[32944]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:31.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:31 smithi050 ceph-mon[32753]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:34.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:33 smithi046 ceph-mon[29201]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:34.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:33 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:34.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:33 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:34.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:33 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:29:34.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:33 smithi050 ceph-mon[32753]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:34.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:33 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:34.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:33 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:34.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:33 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:29:34.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:33 smithi142 ceph-mon[32944]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:34.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:33 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:34.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:33 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:34.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:33 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:29:36.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:35 smithi046 ceph-mon[29201]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:36.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:35 smithi050 ceph-mon[32753]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:36.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:36.216+0000 7f11a5ffb700 1 -- 172.21.15.46:0/1706334670 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f11a8063860 con 0x7f11980a3c40 2024-04-26T21:29:36.217 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 6 on host 'smithi046' 2024-04-26T21:29:36.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:36.218+0000 7f11aec9b700 1 -- 172.21.15.46:0/1706334670 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f11980a3c40 msgr2=0x7f11980a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:36.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:36.218+0000 7f11aec9b700 1 --2- 172.21.15.46:0/1706334670 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f11980a3c40 0x7f11980a6100 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f11a81033e0 tx=0x7f119c00b410 comp rx=0 tx=0).stop 2024-04-26T21:29:36.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:36.218+0000 7f11aec9b700 1 -- 172.21.15.46:0/1706334670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11a8103f20 msgr2=0x7f11a81a0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:36.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:36.218+0000 7f11aec9b700 1 --2- 172.21.15.46:0/1706334670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11a8103f20 0x7f11a81a0930 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f11a000d010 tx=0x7f11a0007a20 comp rx=0 tx=0).stop 2024-04-26T21:29:36.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:36.218+0000 7f11aec9b700 1 -- 172.21.15.46:0/1706334670 shutdown_connections 2024-04-26T21:29:36.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:36.218+0000 7f11aec9b700 1 --2- 172.21.15.46:0/1706334670 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f11980a3c40 0x7f11980a6100 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:36.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:36.218+0000 7f11aec9b700 1 --2- 172.21.15.46:0/1706334670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11a8103f20 0x7f11a81a0930 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:36.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:36.218+0000 7f11aec9b700 1 --2- 172.21.15.46:0/1706334670 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11a8102380 0x7f11a806bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:36.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:36.218+0000 7f11aec9b700 1 --2- 172.21.15.46:0/1706334670 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11a8103580 0x7f11a819e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:36.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:36.218+0000 7f11aec9b700 1 -- 172.21.15.46:0/1706334670 >> 172.21.15.46:0/1706334670 conn(0x7f11a80fd920 msgr2=0x7f11a8106900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:36.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:36.218+0000 7f11aec9b700 1 -- 172.21.15.46:0/1706334670 shutdown_connections 2024-04-26T21:29:36.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:29:36.218+0000 7f11aec9b700 1 -- 172.21.15.46:0/1706334670 wait complete. 2024-04-26T21:29:36.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:35 smithi142 ceph-mon[32944]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:37.199 DEBUG:teuthology.orchestra.run.smithi046:osd.6> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.6.service 2024-04-26T21:29:37.201 INFO:tasks.cephadm:Deploying osd.7 on smithi050 with /dev/vg_nvme/lv_2... 2024-04-26T21:29:37.202 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T21:29:37.457 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T21:29:37.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:37 smithi142 ceph-mon[32944]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:37.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:37 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:37.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:37 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:37.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:37 smithi142 ceph-mon[32944]: from='osd.6 [v2:172.21.15.46:6818/2325162882,v1:172.21.15.46:6819/2325162882]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T21:29:37.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:37 smithi046 ceph-mon[29201]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:37.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:37 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:37.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:37 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:37.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:37 smithi046 ceph-mon[29201]: from='osd.6 [v2:172.21.15.46:6818/2325162882,v1:172.21.15.46:6819/2325162882]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T21:29:37.580 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:37 smithi050 ceph-mon[32753]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:37.580 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:37 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:37.580 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:37 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:37.580 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:37 smithi050 ceph-mon[32753]: from='osd.6 [v2:172.21.15.46:6818/2325162882,v1:172.21.15.46:6819/2325162882]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T21:29:38.288 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:38 smithi046 ceph-mon[29201]: from='osd.6 [v2:172.21.15.46:6818/2325162882,v1:172.21.15.46:6819/2325162882]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T21:29:38.288 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:38 smithi046 ceph-mon[29201]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T21:29:38.288 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:38 smithi046 ceph-mon[29201]: from='osd.6 [v2:172.21.15.46:6818/2325162882,v1:172.21.15.46:6819/2325162882]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-26T21:29:38.288 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:38 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:29:38.289 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:38 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:38.289 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:38 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:38.289 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:29:38 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:29:38.184+0000 7f0c1c37c700 -1 osd.6 0 waiting for initial osdmap 2024-04-26T21:29:38.289 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:29:38 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:29:38.191+0000 7f0c1816f700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T21:29:38.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:38 smithi142 ceph-mon[32944]: from='osd.6 [v2:172.21.15.46:6818/2325162882,v1:172.21.15.46:6819/2325162882]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T21:29:38.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:38 smithi142 ceph-mon[32944]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T21:29:38.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:38 smithi142 ceph-mon[32944]: from='osd.6 [v2:172.21.15.46:6818/2325162882,v1:172.21.15.46:6819/2325162882]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-26T21:29:38.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:38 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:29:38.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:38 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:38.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:38 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:38.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:38 smithi050 ceph-mon[32753]: from='osd.6 [v2:172.21.15.46:6818/2325162882,v1:172.21.15.46:6819/2325162882]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T21:29:38.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:38 smithi050 ceph-mon[32753]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T21:29:38.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:38 smithi050 ceph-mon[32753]: from='osd.6 [v2:172.21.15.46:6818/2325162882,v1:172.21.15.46:6819/2325162882]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-26T21:29:38.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:38 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:29:38.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:38 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:38.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:38 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:39.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:39 smithi046 ceph-mon[29201]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:39.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:39 smithi046 ceph-mon[29201]: from='osd.6 [v2:172.21.15.46:6818/2325162882,v1:172.21.15.46:6819/2325162882]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-26T21:29:39.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:39 smithi046 ceph-mon[29201]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T21:29:39.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:39 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:29:39.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:39 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:29:39.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:39 smithi050 ceph-mon[32753]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:39.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:39 smithi050 ceph-mon[32753]: from='osd.6 [v2:172.21.15.46:6818/2325162882,v1:172.21.15.46:6819/2325162882]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-26T21:29:39.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:39 smithi050 ceph-mon[32753]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T21:29:39.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:39 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:29:39.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:39 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:29:39.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:39 smithi142 ceph-mon[32944]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:29:39.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:39 smithi142 ceph-mon[32944]: from='osd.6 [v2:172.21.15.46:6818/2325162882,v1:172.21.15.46:6819/2325162882]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-26T21:29:39.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:39 smithi142 ceph-mon[32944]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T21:29:39.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:39 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:29:39.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:39 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:29:39.646 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T21:29:39.646 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T21:29:39.646 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10+0 records in 2024-04-26T21:29:39.646 INFO:teuthology.orchestra.run.smithi050.stderr:10+0 records out 2024-04-26T21:29:39.646 INFO:teuthology.orchestra.run.smithi050.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00831298 s, 1.3 GB/s 2024-04-26T21:29:39.646 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping successful for: 2024-04-26T21:29:40.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:40 smithi050 ceph-mon[32753]: purged_snaps scrub starts 2024-04-26T21:29:40.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:40 smithi050 ceph-mon[32753]: purged_snaps scrub ok 2024-04-26T21:29:40.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:40 smithi050 ceph-mon[32753]: osd.6 [v2:172.21.15.46:6818/2325162882,v1:172.21.15.46:6819/2325162882] boot 2024-04-26T21:29:40.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:40 smithi050 ceph-mon[32753]: osdmap e46: 7 total, 7 up, 7 in 2024-04-26T21:29:40.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:40 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:29:40.399 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi050:vg_nvme/lv_2 2024-04-26T21:29:40.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:40 smithi142 ceph-mon[32944]: purged_snaps scrub starts 2024-04-26T21:29:40.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:40 smithi142 ceph-mon[32944]: purged_snaps scrub ok 2024-04-26T21:29:40.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:40 smithi142 ceph-mon[32944]: osd.6 [v2:172.21.15.46:6818/2325162882,v1:172.21.15.46:6819/2325162882] boot 2024-04-26T21:29:40.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:40 smithi142 ceph-mon[32944]: osdmap e46: 7 total, 7 up, 7 in 2024-04-26T21:29:40.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:40 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:29:40.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:40 smithi046 ceph-mon[29201]: purged_snaps scrub starts 2024-04-26T21:29:40.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:40 smithi046 ceph-mon[29201]: purged_snaps scrub ok 2024-04-26T21:29:40.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:40 smithi046 ceph-mon[29201]: osd.6 [v2:172.21.15.46:6818/2325162882,v1:172.21.15.46:6819/2325162882] boot 2024-04-26T21:29:40.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:40 smithi046 ceph-mon[29201]: osdmap e46: 7 total, 7 up, 7 in 2024-04-26T21:29:40.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:40 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:29:40.645 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T21:29:41.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:41 smithi046 ceph-mon[29201]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:41.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:41 smithi046 ceph-mon[29201]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T21:29:41.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:41 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:41.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:41 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:41.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:41 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:29:41.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:41 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:29:41.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:41 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:29:41.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:41 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:41.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:41 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:41.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:41 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:29:41.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:41 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:41.479 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:41 smithi050 ceph-mon[32753]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:41.479 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:41 smithi050 ceph-mon[32753]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T21:29:41.479 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:41 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:41.479 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:41 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:41.479 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:41 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:29:41.479 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:41 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:29:41.479 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:41 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:29:41.480 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:41 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:41.480 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:41 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:41.480 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:41 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:29:41.480 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:41 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:41.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:41 smithi142 ceph-mon[32944]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:41.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:41 smithi142 ceph-mon[32944]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T21:29:41.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:41 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:41.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:41 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:41.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:41 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:29:41.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:41 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:29:41.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:41 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:29:41.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:41 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:41.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:41 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:41.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:41 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:29:41.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:41 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:42.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.335+0000 7fe759c16700 1 -- 172.21.15.50:0/3563886427 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe754105040 msgr2=0x7fe75410b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:42.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.335+0000 7fe759c16700 1 --2- 172.21.15.50:0/3563886427 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe754105040 0x7fe75410b8f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe74c00b0d0 tx=0x7fe74c00b3e0 comp rx=0 tx=0).stop 2024-04-26T21:29:42.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.335+0000 7fe759c16700 1 -- 172.21.15.50:0/3563886427 shutdown_connections 2024-04-26T21:29:42.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.335+0000 7fe759c16700 1 --2- 172.21.15.50:0/3563886427 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7541034a0 0x7fe7541038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:42.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.335+0000 7fe759c16700 1 --2- 172.21.15.50:0/3563886427 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe754105040 0x7fe75410b8f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:42.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.335+0000 7fe759c16700 1 --2- 172.21.15.50:0/3563886427 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7541046a0 0x7fe754104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:42.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.335+0000 7fe759c16700 1 -- 172.21.15.50:0/3563886427 >> 172.21.15.50:0/3563886427 conn(0x7fe7540fea40 msgr2=0x7fe754100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:42.338 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:42 smithi050 ceph-mon[32753]: Detected new or changed devices on smithi046 2024-04-26T21:29:42.338 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:42 smithi050 ceph-mon[32753]: Adjusting osd_memory_target on smithi046 to 5697M 2024-04-26T21:29:42.338 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:42 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:29:42.338 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:42 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:42.338 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:42 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:29:42.338 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:42 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:42.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.335+0000 7fe759c16700 1 -- 172.21.15.50:0/3563886427 shutdown_connections 2024-04-26T21:29:42.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.335+0000 7fe759c16700 1 -- 172.21.15.50:0/3563886427 wait complete. 2024-04-26T21:29:42.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.336+0000 7fe759c16700 1 Processor -- start 2024-04-26T21:29:42.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.336+0000 7fe759c16700 1 -- start start 2024-04-26T21:29:42.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.336+0000 7fe759c16700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7541034a0 0x7fe7540771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:42.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.336+0000 7fe759c16700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7541046a0 0x7fe754075810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:42.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.337+0000 7fe759c16700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe754105040 0x7fe754075d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:42.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.337+0000 7fe759c16700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe75410ef90 con 0x7fe7541046a0 2024-04-26T21:29:42.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.337+0000 7fe759c16700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe75410ee10 con 0x7fe754105040 2024-04-26T21:29:42.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.337+0000 7fe759c16700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe75410f110 con 0x7fe7541034a0 2024-04-26T21:29:42.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.337+0000 7fe753fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe754105040 0x7fe754075d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:42.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.337+0000 7fe753fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe754105040 0x7fe754075d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53322/0 (socket says 172.21.15.50:53322) 2024-04-26T21:29:42.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.337+0000 7fe7537fe700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7541034a0 0x7fe7540771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:42.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.337+0000 7fe753fff700 1 -- 172.21.15.50:0/1921993978 learned_addr learned my addr 172.21.15.50:0/1921993978 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T21:29:42.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.337+0000 7fe752ffd700 1 --2- 172.21.15.50:0/1921993978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7541046a0 0x7fe754075810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:42.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.337+0000 7fe753fff700 1 -- 172.21.15.50:0/1921993978 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7541034a0 msgr2=0x7fe7540771c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:42.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.337+0000 7fe753fff700 1 --2- 172.21.15.50:0/1921993978 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7541034a0 0x7fe7540771c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:42.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.337+0000 7fe753fff700 1 -- 172.21.15.50:0/1921993978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7541046a0 msgr2=0x7fe754075810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:42.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.337+0000 7fe753fff700 1 --2- 172.21.15.50:0/1921993978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7541046a0 0x7fe754075810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:42.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.337+0000 7fe753fff700 1 -- 172.21.15.50:0/1921993978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe754076600 con 0x7fe754105040 2024-04-26T21:29:42.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.337+0000 7fe752ffd700 1 --2- 172.21.15.50:0/1921993978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7541046a0 0x7fe754075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:29:42.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.337+0000 7fe753fff700 1 --2- 172.21.15.50:0/1921993978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe754105040 0x7fe754075d50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe74c009040 tx=0x7fe74c004d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:42.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.337+0000 7fe750ff9700 1 -- 172.21.15.50:0/1921993978 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe74c01f030 con 0x7fe754105040 2024-04-26T21:29:42.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.338+0000 7fe759c16700 1 -- 172.21.15.50:0/1921993978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7541a36b0 con 0x7fe754105040 2024-04-26T21:29:42.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.338+0000 7fe750ff9700 1 -- 172.21.15.50:0/1921993978 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe74c003a00 con 0x7fe754105040 2024-04-26T21:29:42.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.338+0000 7fe750ff9700 1 -- 172.21.15.50:0/1921993978 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe74c0046e0 con 0x7fe754105040 2024-04-26T21:29:42.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.338+0000 7fe759c16700 1 -- 172.21.15.50:0/1921993978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7541a3b00 con 0x7fe754105040 2024-04-26T21:29:42.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.339+0000 7fe759c16700 1 -- 172.21.15.50:0/1921993978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe75406a110 con 0x7fe754105040 2024-04-26T21:29:42.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.340+0000 7fe750ff9700 1 -- 172.21.15.50:0/1921993978 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe74c003bb0 con 0x7fe754105040 2024-04-26T21:29:42.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.340+0000 7fe750ff9700 1 --2- 172.21.15.50:0/1921993978 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe7400a3c40 0x7fe7400a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:42.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.340+0000 7fe750ff9700 1 -- 172.21.15.50:0/1921993978 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7fe74c0d0880 con 0x7fe754105040 2024-04-26T21:29:42.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.341+0000 7fe7537fe700 1 --2- 172.21.15.50:0/1921993978 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe7400a3c40 0x7fe7400a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:42.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.341+0000 7fe7537fe700 1 --2- 172.21.15.50:0/1921993978 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe7400a3c40 0x7fe7400a6100 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe754104500 tx=0x7fe73c007400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:42.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.345+0000 7fe750ff9700 1 -- 172.21.15.50:0/1921993978 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe74c09e000 con 0x7fe754105040 2024-04-26T21:29:42.457 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:42 smithi142 ceph-mon[32944]: Detected new or changed devices on smithi046 2024-04-26T21:29:42.457 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:42 smithi142 ceph-mon[32944]: Adjusting osd_memory_target on smithi046 to 5697M 2024-04-26T21:29:42.457 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:42 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:29:42.457 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:42 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:42.457 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:42 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:29:42.457 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:42 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:42.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:42 smithi046 ceph-mon[29201]: Detected new or changed devices on smithi046 2024-04-26T21:29:42.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:42 smithi046 ceph-mon[29201]: Adjusting osd_memory_target on smithi046 to 5697M 2024-04-26T21:29:42.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:42 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:29:42.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:42 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:42.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:42 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:29:42.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:42 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:42.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:29:42.562+0000 7fe759c16700 1 -- 172.21.15.50:0/1921993978 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fe754063860 con 0x7fe7400a3c40 2024-04-26T21:29:43.529 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:43 smithi050 ceph-mon[32753]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:43.529 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:43 smithi050 ceph-mon[32753]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:29:43.529 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:43 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:29:43.529 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:43 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:29:43.529 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:43 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:43.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:43 smithi142 ceph-mon[32944]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:43.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:43 smithi142 ceph-mon[32944]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:29:43.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:43 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:29:43.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:43 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:29:43.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:43 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:43.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:43 smithi046 ceph-mon[29201]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:43.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:43 smithi046 ceph-mon[29201]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:29:43.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:43 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:29:43.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:43 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:29:43.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:43 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:45.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:45 smithi142 ceph-mon[32944]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:45.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:45 smithi046 ceph-mon[29201]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:45.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:45 smithi050 ceph-mon[32753]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:46.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:46 smithi142 ceph-mon[32944]: from='client.? 172.21.15.50:0/1496198087' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e53ae885-23b6-46fd-b4b1-c62f31a6921f"}]: dispatch 2024-04-26T21:29:46.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:46 smithi142 ceph-mon[32944]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e53ae885-23b6-46fd-b4b1-c62f31a6921f"}]: dispatch 2024-04-26T21:29:46.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:46 smithi142 ceph-mon[32944]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e53ae885-23b6-46fd-b4b1-c62f31a6921f"}]': finished 2024-04-26T21:29:46.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:46 smithi142 ceph-mon[32944]: osdmap e48: 8 total, 7 up, 8 in 2024-04-26T21:29:46.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:46 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:29:46.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:46 smithi142 ceph-mon[32944]: from='client.? 172.21.15.50:0/1349908339' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:29:46.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:46 smithi046 ceph-mon[29201]: from='client.? 172.21.15.50:0/1496198087' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e53ae885-23b6-46fd-b4b1-c62f31a6921f"}]: dispatch 2024-04-26T21:29:46.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:46 smithi046 ceph-mon[29201]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e53ae885-23b6-46fd-b4b1-c62f31a6921f"}]: dispatch 2024-04-26T21:29:46.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:46 smithi046 ceph-mon[29201]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e53ae885-23b6-46fd-b4b1-c62f31a6921f"}]': finished 2024-04-26T21:29:46.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:46 smithi046 ceph-mon[29201]: osdmap e48: 8 total, 7 up, 8 in 2024-04-26T21:29:46.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:46 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:29:46.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:46 smithi046 ceph-mon[29201]: from='client.? 172.21.15.50:0/1349908339' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:29:46.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:46 smithi050 ceph-mon[32753]: from='client.? 172.21.15.50:0/1496198087' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e53ae885-23b6-46fd-b4b1-c62f31a6921f"}]: dispatch 2024-04-26T21:29:46.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:46 smithi050 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e53ae885-23b6-46fd-b4b1-c62f31a6921f"}]: dispatch 2024-04-26T21:29:46.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:46 smithi050 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e53ae885-23b6-46fd-b4b1-c62f31a6921f"}]': finished 2024-04-26T21:29:46.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:46 smithi050 ceph-mon[32753]: osdmap e48: 8 total, 7 up, 8 in 2024-04-26T21:29:46.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:46 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:29:46.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:46 smithi050 ceph-mon[32753]: from='client.? 172.21.15.50:0/1349908339' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:29:47.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:47 smithi050 ceph-mon[32753]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:47.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:47 smithi142 ceph-mon[32944]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:47.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:47 smithi046 ceph-mon[29201]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:49.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:49 smithi050 ceph-mon[32753]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:49.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:49 smithi142 ceph-mon[32944]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:49.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:49 smithi046 ceph-mon[29201]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:51.553 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:51 smithi050 ceph-mon[32753]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:51.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:51 smithi142 ceph-mon[32944]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:51.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:51 smithi046 ceph-mon[29201]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:53.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:53 smithi142 ceph-mon[32944]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:53.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:53 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T21:29:53.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:53 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:53.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:53 smithi142 ceph-mon[32944]: Deploying daemon osd.7 on smithi050 2024-04-26T21:29:53.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:53 smithi050 ceph-mon[32753]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:53.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:53 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T21:29:53.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:53 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:53.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:53 smithi050 ceph-mon[32753]: Deploying daemon osd.7 on smithi050 2024-04-26T21:29:53.623 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:53 smithi046 ceph-mon[29201]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:53.623 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:53 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T21:29:53.623 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:53 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:53.623 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:53 smithi046 ceph-mon[29201]: Deploying daemon osd.7 on smithi050 2024-04-26T21:29:55.756 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:55 smithi050 ceph-mon[32753]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:55.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:55 smithi142 ceph-mon[32944]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:55.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:55 smithi046 ceph-mon[29201]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:57.599 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:57 smithi050 ceph-mon[32753]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:57.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:57 smithi142 ceph-mon[32944]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:57.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:57 smithi046 ceph-mon[29201]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:59.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:59 smithi142 ceph-mon[32944]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:59.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:59 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:59.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:59 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:59.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:29:59 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:29:59.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:59 smithi046 ceph-mon[29201]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:59.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:59 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:59.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:59 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:59.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:29:59 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:29:59.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:59 smithi050 ceph-mon[32753]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:29:59.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:59 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:59.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:59 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:29:59.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:29:59 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:30:00.781 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:00 smithi050 ceph-mon[32753]: overall HEALTH_OK 2024-04-26T21:30:00.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:00 smithi142 ceph-mon[32944]: overall HEALTH_OK 2024-04-26T21:30:00.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:00 smithi046 ceph-mon[29201]: overall HEALTH_OK 2024-04-26T21:30:01.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:01 smithi142 ceph-mon[32944]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:30:01.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:01 smithi046 ceph-mon[29201]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:30:01.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:01 smithi050 ceph-mon[32753]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:30:02.387 INFO:teuthology.orchestra.run.smithi050.stdout:Created osd(s) 7 on host 'smithi050' 2024-04-26T21:30:02.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:30:02.386+0000 7fe750ff9700 1 -- 172.21.15.50:0/1921993978 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe754063860 con 0x7fe7400a3c40 2024-04-26T21:30:02.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:30:02.389+0000 7fe759c16700 1 -- 172.21.15.50:0/1921993978 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe7400a3c40 msgr2=0x7fe7400a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:02.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:30:02.389+0000 7fe759c16700 1 --2- 172.21.15.50:0/1921993978 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe7400a3c40 0x7fe7400a6100 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe754104500 tx=0x7fe73c007400 comp rx=0 tx=0).stop 2024-04-26T21:30:02.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:30:02.389+0000 7fe759c16700 1 -- 172.21.15.50:0/1921993978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe754105040 msgr2=0x7fe754075d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:02.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:30:02.389+0000 7fe759c16700 1 --2- 172.21.15.50:0/1921993978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe754105040 0x7fe754075d50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe74c009040 tx=0x7fe74c004d50 comp rx=0 tx=0).stop 2024-04-26T21:30:02.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:30:02.389+0000 7fe759c16700 1 -- 172.21.15.50:0/1921993978 shutdown_connections 2024-04-26T21:30:02.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:30:02.389+0000 7fe759c16700 1 --2- 172.21.15.50:0/1921993978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe754105040 0x7fe754075d50 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:02.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:30:02.389+0000 7fe759c16700 1 --2- 172.21.15.50:0/1921993978 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe7541034a0 0x7fe7540771c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:02.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:30:02.389+0000 7fe759c16700 1 --2- 172.21.15.50:0/1921993978 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe7400a3c40 0x7fe7400a6100 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:02.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:30:02.389+0000 7fe759c16700 1 --2- 172.21.15.50:0/1921993978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7541046a0 0x7fe754075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:02.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:30:02.389+0000 7fe759c16700 1 -- 172.21.15.50:0/1921993978 >> 172.21.15.50:0/1921993978 conn(0x7fe7540fea40 msgr2=0x7fe7541079c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:02.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:30:02.389+0000 7fe759c16700 1 -- 172.21.15.50:0/1921993978 shutdown_connections 2024-04-26T21:30:02.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:30:02.389+0000 7fe759c16700 1 -- 172.21.15.50:0/1921993978 wait complete. 2024-04-26T21:30:02.768 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:02 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:02.768 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:02 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:02.768 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:02 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:02.768 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:02 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:30:02.768 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:02 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:02.768 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:02 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:02.768 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:02 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:02.768 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:02 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:30:02.769 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:02 smithi050 ceph-mon[32753]: from='osd.7 [v2:172.21.15.50:6816/820310228,v1:172.21.15.50:6817/820310228]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T21:30:02.769 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:02 smithi050 ceph-mon[32753]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T21:30:02.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:02 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:02.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:02 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:02.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:02 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:02.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:02 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:30:02.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:02 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:02.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:02 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:02.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:02 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:02.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:02 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:30:02.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:02 smithi046 ceph-mon[29201]: from='osd.7 [v2:172.21.15.50:6816/820310228,v1:172.21.15.50:6817/820310228]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T21:30:02.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:02 smithi046 ceph-mon[29201]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T21:30:03.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:02 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:03.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:02 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:03.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:02 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:03.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:02 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:30:03.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:02 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:03.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:02 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:03.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:02 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:03.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:02 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:30:03.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:02 smithi142 ceph-mon[32944]: from='osd.7 [v2:172.21.15.50:6816/820310228,v1:172.21.15.50:6817/820310228]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T21:30:03.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:02 smithi142 ceph-mon[32944]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T21:30:03.105 DEBUG:teuthology.orchestra.run.smithi050:osd.7> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.7.service 2024-04-26T21:30:03.107 INFO:tasks.cephadm:Deploying osd.8 on smithi142 with /dev/vg_nvme/lv_2... 2024-04-26T21:30:03.107 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T21:30:03.365 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:30:03.593 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:03 smithi050 ceph-mon[32753]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:30:03.593 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:03 smithi050 ceph-mon[32753]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T21:30:03.593 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:03 smithi050 ceph-mon[32753]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T21:30:03.593 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:03 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:30:03.593 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:03 smithi050 ceph-mon[32753]: from='osd.7 [v2:172.21.15.50:6816/820310228,v1:172.21.15.50:6817/820310228]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T21:30:03.593 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:03 smithi050 ceph-mon[32753]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T21:30:03.772 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:03 smithi142 ceph-mon[32944]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:30:03.772 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:03 smithi142 ceph-mon[32944]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T21:30:03.772 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:03 smithi142 ceph-mon[32944]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T21:30:03.772 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:03 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:30:03.772 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:03 smithi142 ceph-mon[32944]: from='osd.7 [v2:172.21.15.50:6816/820310228,v1:172.21.15.50:6817/820310228]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T21:30:03.772 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:03 smithi142 ceph-mon[32944]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T21:30:03.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:03 smithi046 ceph-mon[29201]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:30:03.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:03 smithi046 ceph-mon[29201]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T21:30:03.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:03 smithi046 ceph-mon[29201]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T21:30:03.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:03 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:30:03.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:03 smithi046 ceph-mon[29201]: from='osd.7 [v2:172.21.15.50:6816/820310228,v1:172.21.15.50:6817/820310228]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T21:30:03.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:03 smithi046 ceph-mon[29201]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T21:30:04.618 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:30:04 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:30:04.354+0000 7fb1e8528700 -1 osd.7 0 waiting for initial osdmap 2024-04-26T21:30:04.618 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:30:04 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:30:04.365+0000 7fb1e431b700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T21:30:05.415 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T21:30:05.415 INFO:teuthology.orchestra.run.smithi142.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T21:30:05.415 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10+0 records in 2024-04-26T21:30:05.415 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records out 2024-04-26T21:30:05.415 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131531 s, 797 MB/s 2024-04-26T21:30:05.415 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping successful for: 2024-04-26T21:30:05.563 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:05 smithi142 ceph-mon[32944]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:30:05.563 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:05 smithi142 ceph-mon[32944]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T21:30:05.563 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:05 smithi142 ceph-mon[32944]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T21:30:05.563 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:05 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:30:05.563 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:05 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:30:05.563 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:05 smithi142 ceph-mon[32944]: osd.7 [v2:172.21.15.50:6816/820310228,v1:172.21.15.50:6817/820310228] boot 2024-04-26T21:30:05.563 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:05 smithi142 ceph-mon[32944]: osdmap e51: 8 total, 8 up, 8 in 2024-04-26T21:30:05.563 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:05 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:30:05.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:05 smithi050 ceph-mon[32753]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:30:05.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:05 smithi050 ceph-mon[32753]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T21:30:05.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:05 smithi050 ceph-mon[32753]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T21:30:05.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:05 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:30:05.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:05 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:30:05.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:05 smithi050 ceph-mon[32753]: osd.7 [v2:172.21.15.50:6816/820310228,v1:172.21.15.50:6817/820310228] boot 2024-04-26T21:30:05.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:05 smithi050 ceph-mon[32753]: osdmap e51: 8 total, 8 up, 8 in 2024-04-26T21:30:05.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:05 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:30:05.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:05 smithi046 ceph-mon[29201]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:30:05.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:05 smithi046 ceph-mon[29201]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T21:30:05.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:05 smithi046 ceph-mon[29201]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T21:30:05.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:05 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:30:05.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:05 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:30:05.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:05 smithi046 ceph-mon[29201]: osd.7 [v2:172.21.15.50:6816/820310228,v1:172.21.15.50:6817/820310228] boot 2024-04-26T21:30:05.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:05 smithi046 ceph-mon[29201]: osdmap e51: 8 total, 8 up, 8 in 2024-04-26T21:30:05.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:05 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:30:06.119 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi142:vg_nvme/lv_2 2024-04-26T21:30:06.366 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:30:06.616 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:06 smithi142 ceph-mon[32944]: purged_snaps scrub starts 2024-04-26T21:30:06.616 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:06 smithi142 ceph-mon[32944]: purged_snaps scrub ok 2024-04-26T21:30:06.616 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:06 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:06.616 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:06 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:06.616 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:06 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:06.616 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:06 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:06.616 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:06 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:06.617 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:06 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:06.617 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:06 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:06.617 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:06 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:30:06.617 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:06 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:06.617 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:06 smithi142 ceph-mon[32944]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T21:30:06.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:06 smithi050 ceph-mon[32753]: purged_snaps scrub starts 2024-04-26T21:30:06.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:06 smithi050 ceph-mon[32753]: purged_snaps scrub ok 2024-04-26T21:30:06.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:06 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:06.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:06 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:06.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:06 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:06.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:06 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:06.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:06 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:06.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:06 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:06.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:06 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:06.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:06 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:30:06.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:06 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:06.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:06 smithi050 ceph-mon[32753]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T21:30:06.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:06 smithi046 ceph-mon[29201]: purged_snaps scrub starts 2024-04-26T21:30:06.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:06 smithi046 ceph-mon[29201]: purged_snaps scrub ok 2024-04-26T21:30:06.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:06 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:06.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:06 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:06.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:06 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:06.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:06 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:06.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:06 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:06.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:06 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:06.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:06 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:06.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:06 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:30:06.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:06 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:06.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:06 smithi046 ceph-mon[29201]: osdmap e52: 8 total, 8 up, 8 in 2024-04-26T21:30:07.456 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:07 smithi142 ceph-mon[32944]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:07.457 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:07 smithi142 ceph-mon[32944]: Detected new or changed devices on smithi050 2024-04-26T21:30:07.457 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:07 smithi142 ceph-mon[32944]: Adjusting osd_memory_target on smithi050 to 5697M 2024-04-26T21:30:07.457 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:07 smithi142 ceph-mon[32944]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T21:30:07.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:07 smithi050 ceph-mon[32753]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:07.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:07 smithi050 ceph-mon[32753]: Detected new or changed devices on smithi050 2024-04-26T21:30:07.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:07 smithi050 ceph-mon[32753]: Adjusting osd_memory_target on smithi050 to 5697M 2024-04-26T21:30:07.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:07 smithi050 ceph-mon[32753]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T21:30:07.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:07 smithi046 ceph-mon[29201]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:07.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:07 smithi046 ceph-mon[29201]: Detected new or changed devices on smithi050 2024-04-26T21:30:07.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:07 smithi046 ceph-mon[29201]: Adjusting osd_memory_target on smithi050 to 5697M 2024-04-26T21:30:07.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:07 smithi046 ceph-mon[29201]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T21:30:07.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.954+0000 7ffb39236700 1 -- 172.21.15.142:0/406318921 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb34102380 msgr2=0x7ffb341027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.954+0000 7ffb39236700 1 --2- 172.21.15.142:0/406318921 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb34102380 0x7ffb341027a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ffb24009990 tx=0x7ffb24009ca0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.954+0000 7ffb39236700 1 -- 172.21.15.142:0/406318921 shutdown_connections 2024-04-26T21:30:07.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.954+0000 7ffb39236700 1 --2- 172.21.15.142:0/406318921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb34103f20 0x7ffb3410a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.954+0000 7ffb39236700 1 --2- 172.21.15.142:0/406318921 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb34102380 0x7ffb341027a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.954+0000 7ffb39236700 1 --2- 172.21.15.142:0/406318921 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb34103580 0x7ffb341039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.954+0000 7ffb39236700 1 -- 172.21.15.142:0/406318921 >> 172.21.15.142:0/406318921 conn(0x7ffb340fd920 msgr2=0x7ffb340ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:07.957 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.954+0000 7ffb39236700 1 -- 172.21.15.142:0/406318921 shutdown_connections 2024-04-26T21:30:07.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.954+0000 7ffb39236700 1 -- 172.21.15.142:0/406318921 wait complete. 2024-04-26T21:30:07.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.955+0000 7ffb39236700 1 Processor -- start 2024-04-26T21:30:07.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.955+0000 7ffb39236700 1 -- start start 2024-04-26T21:30:07.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.955+0000 7ffb39236700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb34102380 0x7ffb3406bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.955+0000 7ffb39236700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb34103580 0x7ffb3419e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.955+0000 7ffb39236700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb34103f20 0x7ffb341a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.955+0000 7ffb39236700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ffb3410ddf0 con 0x7ffb34103f20 2024-04-26T21:30:07.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.955+0000 7ffb39236700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ffb3410dc70 con 0x7ffb34102380 2024-04-26T21:30:07.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.955+0000 7ffb39236700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ffb3410df70 con 0x7ffb34103580 2024-04-26T21:30:07.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.955+0000 7ffb2a59c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb34103580 0x7ffb3419e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.955+0000 7ffb2a59c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb34103580 0x7ffb3419e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:35232/0 (socket says 172.21.15.142:35232) 2024-04-26T21:30:07.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.955+0000 7ffb2a59c700 1 -- 172.21.15.142:0/3416049721 learned_addr learned my addr 172.21.15.142:0/3416049721 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-04-26T21:30:07.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.955+0000 7ffb32d9d700 1 --2- 172.21.15.142:0/3416049721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb34102380 0x7ffb3406bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.958 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.955+0000 7ffb3359e700 1 --2- 172.21.15.142:0/3416049721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb34103f20 0x7ffb341a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.955+0000 7ffb2a59c700 1 -- 172.21.15.142:0/3416049721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb34102380 msgr2=0x7ffb3406bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.955+0000 7ffb2a59c700 1 --2- 172.21.15.142:0/3416049721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb34102380 0x7ffb3406bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.955+0000 7ffb2a59c700 1 -- 172.21.15.142:0/3416049721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb34103f20 msgr2=0x7ffb341a09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.955+0000 7ffb2a59c700 1 --2- 172.21.15.142:0/3416049721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb34103f20 0x7ffb341a09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.956+0000 7ffb2a59c700 1 -- 172.21.15.142:0/3416049721 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb341a12b0 con 0x7ffb34103580 2024-04-26T21:30:07.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.956+0000 7ffb2a59c700 1 --2- 172.21.15.142:0/3416049721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb34103580 0x7ffb3419e3a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c00d950 tx=0x7ffb1c00dd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:07.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.956+0000 7ffb30d99700 1 -- 172.21.15.142:0/3416049721 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb1c016070 con 0x7ffb34103580 2024-04-26T21:30:07.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.956+0000 7ffb30d99700 1 -- 172.21.15.142:0/3416049721 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb1c010460 con 0x7ffb34103580 2024-04-26T21:30:07.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.956+0000 7ffb39236700 1 -- 172.21.15.142:0/3416049721 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb341a15a0 con 0x7ffb34103580 2024-04-26T21:30:07.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.956+0000 7ffb30d99700 1 -- 172.21.15.142:0/3416049721 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb1c00f620 con 0x7ffb34103580 2024-04-26T21:30:07.959 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.956+0000 7ffb39236700 1 -- 172.21.15.142:0/3416049721 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb341a1b30 con 0x7ffb34103580 2024-04-26T21:30:07.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.957+0000 7ffb39236700 1 -- 172.21.15.142:0/3416049721 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb3406a110 con 0x7ffb34103580 2024-04-26T21:30:07.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.958+0000 7ffb30d99700 1 -- 172.21.15.142:0/3416049721 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ffb1c019070 con 0x7ffb34103580 2024-04-26T21:30:07.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.959+0000 7ffb30d99700 1 --2- 172.21.15.142:0/3416049721 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ffb200a3cc0 0x7ffb200a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.960 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.959+0000 7ffb30d99700 1 -- 172.21.15.142:0/3416049721 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7ffb1c0cf530 con 0x7ffb34103580 2024-04-26T21:30:07.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.959+0000 7ffb32d9d700 1 --2- 172.21.15.142:0/3416049721 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ffb200a3cc0 0x7ffb200a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.961 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.960+0000 7ffb32d9d700 1 --2- 172.21.15.142:0/3416049721 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ffb200a3cc0 0x7ffb200a6180 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ffb2400b5c0 tx=0x7ffb24005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:07.963 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:07.963+0000 7ffb30d99700 1 -- 172.21.15.142:0/3416049721 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ffb1c09c9b0 con 0x7ffb34103580 2024-04-26T21:30:08.171 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:08.170+0000 7ffb39236700 1 -- 172.21.15.142:0/3416049721 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7ffb34063860 con 0x7ffb200a3cc0 2024-04-26T21:30:08.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:08 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:30:08.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:08 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:30:08.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:08 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:08.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:08 smithi142 ceph-mon[32944]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T21:30:08.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:08 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:30:08.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:08 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:30:08.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:08 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:08.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:08 smithi050 ceph-mon[32753]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T21:30:08.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:08 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:30:08.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:08 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:30:08.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:08 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:08.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:08 smithi046 ceph-mon[29201]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T21:30:09.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:09 smithi142 ceph-mon[32944]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:09.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:09 smithi142 ceph-mon[32944]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:30:09.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:09 smithi142 ceph-mon[32944]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T21:30:09.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:09 smithi050 ceph-mon[32753]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:09.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:09 smithi050 ceph-mon[32753]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:30:09.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:09 smithi050 ceph-mon[32753]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T21:30:09.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:09 smithi046 ceph-mon[29201]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:09.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:09 smithi046 ceph-mon[29201]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:30:09.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:09 smithi046 ceph-mon[29201]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T21:30:11.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:11 smithi142 ceph-mon[32944]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:11.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:11 smithi142 ceph-mon[32944]: from='client.? 172.21.15.142:0/2399019289' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "356e2692-16a9-4045-b58f-c8f9cb5d7241"}]: dispatch 2024-04-26T21:30:11.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:11 smithi142 ceph-mon[32944]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "356e2692-16a9-4045-b58f-c8f9cb5d7241"}]: dispatch 2024-04-26T21:30:11.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:11 smithi142 ceph-mon[32944]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "356e2692-16a9-4045-b58f-c8f9cb5d7241"}]': finished 2024-04-26T21:30:11.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:11 smithi142 ceph-mon[32944]: osdmap e56: 9 total, 8 up, 9 in 2024-04-26T21:30:11.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:11 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:30:11.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:11 smithi050 ceph-mon[32753]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:11.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:11 smithi050 ceph-mon[32753]: from='client.? 172.21.15.142:0/2399019289' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "356e2692-16a9-4045-b58f-c8f9cb5d7241"}]: dispatch 2024-04-26T21:30:11.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:11 smithi050 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "356e2692-16a9-4045-b58f-c8f9cb5d7241"}]: dispatch 2024-04-26T21:30:11.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:11 smithi050 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "356e2692-16a9-4045-b58f-c8f9cb5d7241"}]': finished 2024-04-26T21:30:11.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:11 smithi050 ceph-mon[32753]: osdmap e56: 9 total, 8 up, 9 in 2024-04-26T21:30:11.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:11 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:30:11.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:11 smithi046 ceph-mon[29201]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:11.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:11 smithi046 ceph-mon[29201]: from='client.? 172.21.15.142:0/2399019289' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "356e2692-16a9-4045-b58f-c8f9cb5d7241"}]: dispatch 2024-04-26T21:30:11.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:11 smithi046 ceph-mon[29201]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "356e2692-16a9-4045-b58f-c8f9cb5d7241"}]: dispatch 2024-04-26T21:30:11.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:11 smithi046 ceph-mon[29201]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "356e2692-16a9-4045-b58f-c8f9cb5d7241"}]': finished 2024-04-26T21:30:11.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:11 smithi046 ceph-mon[29201]: osdmap e56: 9 total, 8 up, 9 in 2024-04-26T21:30:11.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:11 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:30:12.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:12 smithi050 ceph-mon[32753]: from='client.? 172.21.15.142:0/3677615441' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:30:12.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:12 smithi142 ceph-mon[32944]: from='client.? 172.21.15.142:0/3677615441' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:30:12.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:12 smithi046 ceph-mon[29201]: from='client.? 172.21.15.142:0/3677615441' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:30:13.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:13 smithi142 ceph-mon[32944]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:13.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:13 smithi046 ceph-mon[29201]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:13.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:13 smithi050 ceph-mon[32753]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:15.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:15 smithi142 ceph-mon[32944]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:15.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:15 smithi046 ceph-mon[29201]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:15.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:15 smithi050 ceph-mon[32753]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:17.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:17 smithi142 ceph-mon[32944]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:17.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:17 smithi046 ceph-mon[29201]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:17.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:17 smithi050 ceph-mon[32753]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:18.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:18 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T21:30:18.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:18 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:18.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:18 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T21:30:18.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:18 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:18.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:18 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T21:30:18.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:18 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:19.729 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:19 smithi142 ceph-mon[32944]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:19.729 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:19 smithi142 ceph-mon[32944]: Deploying daemon osd.8 on smithi142 2024-04-26T21:30:19.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:19 smithi046 ceph-mon[29201]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:19.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:19 smithi046 ceph-mon[29201]: Deploying daemon osd.8 on smithi142 2024-04-26T21:30:19.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:19 smithi050 ceph-mon[32753]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:19.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:19 smithi050 ceph-mon[32753]: Deploying daemon osd.8 on smithi142 2024-04-26T21:30:21.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:21 smithi142 ceph-mon[32944]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:21.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:21 smithi046 ceph-mon[29201]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:21.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:21 smithi050 ceph-mon[32753]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:23.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:23 smithi142 ceph-mon[32944]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:23.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:23 smithi046 ceph-mon[29201]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:23.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:23 smithi050 ceph-mon[32753]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:24 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:24 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:24 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:30:25.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:24 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:25.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:24 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:25.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:24 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:30:25.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:24 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:25.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:24 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:25.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:24 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:30:26.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:25 smithi142 ceph-mon[32944]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:26.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:25 smithi046 ceph-mon[29201]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:26.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:25 smithi050 ceph-mon[32753]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:27.589 INFO:teuthology.orchestra.run.smithi142.stdout:Created osd(s) 8 on host 'smithi142' 2024-04-26T21:30:27.590 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:27.588+0000 7ffb30d99700 1 -- 172.21.15.142:0/3416049721 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ffb34063860 con 0x7ffb200a3cc0 2024-04-26T21:30:27.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:27.591+0000 7ffb39236700 1 -- 172.21.15.142:0/3416049721 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ffb200a3cc0 msgr2=0x7ffb200a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:27.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:27.591+0000 7ffb39236700 1 --2- 172.21.15.142:0/3416049721 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ffb200a3cc0 0x7ffb200a6180 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ffb2400b5c0 tx=0x7ffb24005fd0 comp rx=0 tx=0).stop 2024-04-26T21:30:27.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:27.591+0000 7ffb39236700 1 -- 172.21.15.142:0/3416049721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb34103580 msgr2=0x7ffb3419e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:27.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:27.591+0000 7ffb39236700 1 --2- 172.21.15.142:0/3416049721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb34103580 0x7ffb3419e3a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c00d950 tx=0x7ffb1c00dd10 comp rx=0 tx=0).stop 2024-04-26T21:30:27.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:27.591+0000 7ffb39236700 1 -- 172.21.15.142:0/3416049721 shutdown_connections 2024-04-26T21:30:27.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:27.591+0000 7ffb39236700 1 --2- 172.21.15.142:0/3416049721 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ffb200a3cc0 0x7ffb200a6180 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:27.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:27.591+0000 7ffb39236700 1 --2- 172.21.15.142:0/3416049721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb34103f20 0x7ffb341a09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:27.592 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:27.591+0000 7ffb39236700 1 --2- 172.21.15.142:0/3416049721 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffb34102380 0x7ffb3406bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:27.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:27.591+0000 7ffb39236700 1 --2- 172.21.15.142:0/3416049721 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffb34103580 0x7ffb3419e3a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:27.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:27.591+0000 7ffb39236700 1 -- 172.21.15.142:0/3416049721 >> 172.21.15.142:0/3416049721 conn(0x7ffb340fd920 msgr2=0x7ffb34106a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:27.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:27.591+0000 7ffb39236700 1 -- 172.21.15.142:0/3416049721 shutdown_connections 2024-04-26T21:30:27.593 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:30:27.591+0000 7ffb39236700 1 -- 172.21.15.142:0/3416049721 wait complete. 2024-04-26T21:30:27.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:27 smithi050 ceph-mon[32753]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:27.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:27 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:27.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:27 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:27.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:27 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:27.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:27 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:30:27.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:27 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:27.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:27 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:27.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:27 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:27.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:27 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:30:27.903 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:27 smithi142 ceph-mon[32944]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:27.903 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:27 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:27.904 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:27 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:27.904 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:27 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:27.904 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:27 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:30:27.904 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:27 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:27.904 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:27 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:27.904 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:27 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:27.904 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:27 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:30:28.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:27 smithi046 ceph-mon[29201]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:28.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:27 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:28.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:27 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:28.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:27 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:28.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:27 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:30:28.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:27 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:28.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:27 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:28.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:27 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:28.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:27 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:30:28.269 DEBUG:teuthology.orchestra.run.smithi142:osd.8> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.8.service 2024-04-26T21:30:28.271 INFO:tasks.cephadm:Deploying osd.9 on smithi046 with /dev/vg_nvme/lv_1... 2024-04-26T21:30:28.272 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T21:30:28.528 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:30:28.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:28 smithi050 ceph-mon[32753]: from='osd.8 [v2:172.21.15.142:6816/1824918711,v1:172.21.15.142:6817/1824918711]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T21:30:28.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:28 smithi050 ceph-mon[32753]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T21:30:28.905 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:28 smithi046 ceph-mon[29201]: from='osd.8 [v2:172.21.15.142:6816/1824918711,v1:172.21.15.142:6817/1824918711]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T21:30:28.906 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:28 smithi046 ceph-mon[29201]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T21:30:29.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:28 smithi142 ceph-mon[32944]: from='osd.8 [v2:172.21.15.142:6816/1824918711,v1:172.21.15.142:6817/1824918711]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T21:30:29.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:28 smithi142 ceph-mon[32944]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T21:30:29.730 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:29 smithi142 ceph-mon[32944]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:29.730 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:29 smithi142 ceph-mon[32944]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T21:30:29.730 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:29 smithi142 ceph-mon[32944]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T21:30:29.730 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:29 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:30:29.730 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:29 smithi142 ceph-mon[32944]: from='osd.8 [v2:172.21.15.142:6816/1824918711,v1:172.21.15.142:6817/1824918711]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-04-26T21:30:29.730 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:29 smithi142 ceph-mon[32944]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-04-26T21:30:29.964 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:29 smithi046 ceph-mon[29201]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:29.965 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:29 smithi046 ceph-mon[29201]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T21:30:29.965 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:29 smithi046 ceph-mon[29201]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T21:30:29.965 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:29 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:30:29.965 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:29 smithi046 ceph-mon[29201]: from='osd.8 [v2:172.21.15.142:6816/1824918711,v1:172.21.15.142:6817/1824918711]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-04-26T21:30:29.965 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:29 smithi046 ceph-mon[29201]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-04-26T21:30:30.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:29 smithi050 ceph-mon[32753]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:30.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:29 smithi050 ceph-mon[32753]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T21:30:30.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:29 smithi050 ceph-mon[32753]: osdmap e57: 9 total, 8 up, 9 in 2024-04-26T21:30:30.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:29 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:30:30.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:29 smithi050 ceph-mon[32753]: from='osd.8 [v2:172.21.15.142:6816/1824918711,v1:172.21.15.142:6817/1824918711]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-04-26T21:30:30.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:29 smithi050 ceph-mon[32753]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-04-26T21:30:30.541 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:30:30 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:30:30.081+0000 7fbe7c37f700 -1 osd.8 0 waiting for initial osdmap 2024-04-26T21:30:30.541 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:30:30 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:30:30.093+0000 7fbe7416a700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T21:30:30.715 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T21:30:30.715 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T21:30:30.715 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2024-04-26T21:30:30.715 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2024-04-26T21:30:30.715 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00853326 s, 1.2 GB/s 2024-04-26T21:30:30.715 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2024-04-26T21:30:30.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:30 smithi046 ceph-mon[29201]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-04-26T21:30:30.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:30 smithi046 ceph-mon[29201]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T21:30:30.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:30 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:30:30.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:30 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:30:30.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:30 smithi046 ceph-mon[29201]: from='osd.8 ' entity='osd.8' 2024-04-26T21:30:30.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:30 smithi046 ceph-mon[29201]: osd.8 [v2:172.21.15.142:6816/1824918711,v1:172.21.15.142:6817/1824918711] boot 2024-04-26T21:30:30.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:30 smithi046 ceph-mon[29201]: osdmap e59: 9 total, 9 up, 9 in 2024-04-26T21:30:30.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:30 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:30:30.910 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:30 smithi142 ceph-mon[32944]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-04-26T21:30:30.910 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:30 smithi142 ceph-mon[32944]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T21:30:30.910 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:30 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:30:30.911 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:30 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:30:30.911 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:30 smithi142 ceph-mon[32944]: from='osd.8 ' entity='osd.8' 2024-04-26T21:30:30.911 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:30 smithi142 ceph-mon[32944]: osd.8 [v2:172.21.15.142:6816/1824918711,v1:172.21.15.142:6817/1824918711] boot 2024-04-26T21:30:30.911 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:30 smithi142 ceph-mon[32944]: osdmap e59: 9 total, 9 up, 9 in 2024-04-26T21:30:30.911 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:30 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:30:31.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:30 smithi050 ceph-mon[32753]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-04-26T21:30:31.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:30 smithi050 ceph-mon[32753]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T21:30:31.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:30 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:30:31.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:30 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:30:31.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:30 smithi050 ceph-mon[32753]: from='osd.8 ' entity='osd.8' 2024-04-26T21:30:31.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:30 smithi050 ceph-mon[32753]: osd.8 [v2:172.21.15.142:6816/1824918711,v1:172.21.15.142:6817/1824918711] boot 2024-04-26T21:30:31.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:30 smithi050 ceph-mon[32753]: osdmap e59: 9 total, 9 up, 9 in 2024-04-26T21:30:31.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:30 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:30:31.537 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi046:vg_nvme/lv_1 2024-04-26T21:30:31.782 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:30:31.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:31 smithi046 ceph-mon[29201]: purged_snaps scrub starts 2024-04-26T21:30:31.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:31 smithi046 ceph-mon[29201]: purged_snaps scrub ok 2024-04-26T21:30:31.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:31 smithi046 ceph-mon[29201]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:31.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:31.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:31.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:31.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:31.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:31.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:31.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:31.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:30:31.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:31.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:31 smithi046 ceph-mon[29201]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T21:30:32.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:31 smithi142 ceph-mon[32944]: purged_snaps scrub starts 2024-04-26T21:30:32.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:31 smithi142 ceph-mon[32944]: purged_snaps scrub ok 2024-04-26T21:30:32.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:31 smithi142 ceph-mon[32944]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:32.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:32.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:32.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:32.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:32.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:32.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:32.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:32.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:30:32.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:32.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:31 smithi142 ceph-mon[32944]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T21:30:32.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:31 smithi050 ceph-mon[32753]: purged_snaps scrub starts 2024-04-26T21:30:32.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:31 smithi050 ceph-mon[32753]: purged_snaps scrub ok 2024-04-26T21:30:32.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:31 smithi050 ceph-mon[32753]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 248 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:30:32.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:32.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:32.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:32.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:32.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:32.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:32.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:32.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:30:32.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:32.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:31 smithi050 ceph-mon[32753]: osdmap e60: 9 total, 9 up, 9 in 2024-04-26T21:30:33.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:32 smithi142 ceph-mon[32944]: Detected new or changed devices on smithi142 2024-04-26T21:30:33.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:32 smithi142 ceph-mon[32944]: Adjusting osd_memory_target on smithi142 to 5696M 2024-04-26T21:30:33.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:32 smithi142 ceph-mon[32944]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T21:30:33.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:32 smithi046 ceph-mon[29201]: Detected new or changed devices on smithi142 2024-04-26T21:30:33.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:32 smithi046 ceph-mon[29201]: Adjusting osd_memory_target on smithi142 to 5696M 2024-04-26T21:30:33.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:32 smithi046 ceph-mon[29201]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T21:30:33.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:32 smithi050 ceph-mon[32753]: Detected new or changed devices on smithi142 2024-04-26T21:30:33.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:32 smithi050 ceph-mon[32753]: Adjusting osd_memory_target on smithi142 to 5696M 2024-04-26T21:30:33.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:32 smithi050 ceph-mon[32753]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T21:30:33.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.641+0000 7f42e6a54700 1 -- 172.21.15.46:0/2558755059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42e01091d0 msgr2=0x7f42e010b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:33.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.641+0000 7f42e6a54700 1 --2- 172.21.15.46:0/2558755059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42e01091d0 0x7f42e010b5e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f42d800b0a0 tx=0x7f42d800b3b0 comp rx=0 tx=0).stop 2024-04-26T21:30:33.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.642+0000 7f42e6a54700 1 -- 172.21.15.46:0/2558755059 shutdown_connections 2024-04-26T21:30:33.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.642+0000 7f42e6a54700 1 --2- 172.21.15.46:0/2558755059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42e01091d0 0x7f42e010b5e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:33.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.642+0000 7f42e6a54700 1 --2- 172.21.15.46:0/2558755059 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f42e0103f70 0x7f42e0106360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:33.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.642+0000 7f42e6a54700 1 --2- 172.21.15.46:0/2558755059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42e01068a0 0x7f42e0108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:33.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.642+0000 7f42e6a54700 1 -- 172.21.15.46:0/2558755059 >> 172.21.15.46:0/2558755059 conn(0x7f42e00fd920 msgr2=0x7f42e00ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:33.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.642+0000 7f42e6a54700 1 -- 172.21.15.46:0/2558755059 shutdown_connections 2024-04-26T21:30:33.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.642+0000 7f42e6a54700 1 -- 172.21.15.46:0/2558755059 wait complete. 2024-04-26T21:30:33.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.642+0000 7f42e6a54700 1 Processor -- start 2024-04-26T21:30:33.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.642+0000 7f42e6a54700 1 -- start start 2024-04-26T21:30:33.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.643+0000 7f42e6a54700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f42e0103f70 0x7f42e01995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:33.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.643+0000 7f42e6a54700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42e01068a0 0x7f42e0199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:33.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.643+0000 7f42e6a54700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42e01091d0 0x7f42e01a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:33.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.643+0000 7f42e6a54700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f42e010ef90 con 0x7f42e01068a0 2024-04-26T21:30:33.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.643+0000 7f42e6a54700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f42e010ee10 con 0x7f42e01091d0 2024-04-26T21:30:33.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.643+0000 7f42e6a54700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f42e010f110 con 0x7f42e0103f70 2024-04-26T21:30:33.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.643+0000 7f42df7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42e01068a0 0x7f42e0199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:33.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.643+0000 7f42df7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42e01068a0 0x7f42e0199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54974/0 (socket says 172.21.15.46:54974) 2024-04-26T21:30:33.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.643+0000 7f42df7fe700 1 -- 172.21.15.46:0/13601090 learned_addr learned my addr 172.21.15.46:0/13601090 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:30:33.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.643+0000 7f42dffff700 1 --2- 172.21.15.46:0/13601090 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f42e0103f70 0x7f42e01995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:33.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.643+0000 7f42e4ff1700 1 --2- 172.21.15.46:0/13601090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42e01091d0 0x7f42e01a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:33.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.643+0000 7f42df7fe700 1 -- 172.21.15.46:0/13601090 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f42e0103f70 msgr2=0x7f42e01995d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:33.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.643+0000 7f42df7fe700 1 --2- 172.21.15.46:0/13601090 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f42e0103f70 0x7f42e01995d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:33.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.643+0000 7f42df7fe700 1 -- 172.21.15.46:0/13601090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42e01091d0 msgr2=0x7f42e01a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:33.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.643+0000 7f42df7fe700 1 --2- 172.21.15.46:0/13601090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42e01091d0 0x7f42e01a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:33.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.643+0000 7f42df7fe700 1 -- 172.21.15.46:0/13601090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42e01a1230 con 0x7f42e01068a0 2024-04-26T21:30:33.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.643+0000 7f42df7fe700 1 --2- 172.21.15.46:0/13601090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42e01068a0 0x7f42e0199b10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f42d400eb60 tx=0x7f42d400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:33.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.643+0000 7f42dd7fa700 1 -- 172.21.15.46:0/13601090 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d400cbd0 con 0x7f42e01068a0 2024-04-26T21:30:33.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.643+0000 7f42e6a54700 1 -- 172.21.15.46:0/13601090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42e01a1520 con 0x7f42e01068a0 2024-04-26T21:30:33.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.643+0000 7f42dd7fa700 1 -- 172.21.15.46:0/13601090 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42d40044d0 con 0x7f42e01068a0 2024-04-26T21:30:33.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.644+0000 7f42dd7fa700 1 -- 172.21.15.46:0/13601090 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d4010430 con 0x7f42e01068a0 2024-04-26T21:30:33.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.644+0000 7f42e6a54700 1 -- 172.21.15.46:0/13601090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42e01a1a60 con 0x7f42e01068a0 2024-04-26T21:30:33.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.644+0000 7f42e6a54700 1 -- 172.21.15.46:0/13601090 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42e006a110 con 0x7f42e01068a0 2024-04-26T21:30:33.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.646+0000 7f42dd7fa700 1 -- 172.21.15.46:0/13601090 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f42d4003e70 con 0x7f42e01068a0 2024-04-26T21:30:33.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.646+0000 7f42dd7fa700 1 --2- 172.21.15.46:0/13601090 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f42c80a3c40 0x7f42c80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:33.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.647+0000 7f42dffff700 1 --2- 172.21.15.46:0/13601090 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f42c80a3c40 0x7f42c80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:33.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.647+0000 7f42dd7fa700 1 -- 172.21.15.46:0/13601090 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5829+0+0 (secure 0 0 0) 0x7f42d40cfc10 con 0x7f42e01068a0 2024-04-26T21:30:33.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.647+0000 7f42dffff700 1 --2- 172.21.15.46:0/13601090 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f42c80a3c40 0x7f42c80a6100 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f42d0009c60 tx=0x7f42d0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:33.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.651+0000 7f42dd7fa700 1 -- 172.21.15.46:0/13601090 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f42d409cfa0 con 0x7f42e01068a0 2024-04-26T21:30:33.653 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:33 smithi046 ceph-mon[29201]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:30:33.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:33.871+0000 7f42e6a54700 1 -- 172.21.15.46:0/13601090 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f42e0063860 con 0x7f42c80a3c40 2024-04-26T21:30:34.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:33 smithi142 ceph-mon[32944]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:30:34.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:33 smithi050 ceph-mon[32753]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:30:34.775 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:34 smithi046 ceph-mon[29201]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T21:30:34.775 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:34 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:30:34.775 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:34 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:30:34.775 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:34 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:35.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:34 smithi142 ceph-mon[32944]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T21:30:35.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:34 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:30:35.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:34 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:30:35.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:34 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:35.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:34 smithi050 ceph-mon[32753]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T21:30:35.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:34 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:30:35.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:34 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:30:35.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:34 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:35.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:35 smithi046 ceph-mon[29201]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:30:35.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:35 smithi046 ceph-mon[29201]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:30:35.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:35 smithi046 ceph-mon[29201]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T21:30:36.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:35 smithi142 ceph-mon[32944]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:30:36.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:35 smithi142 ceph-mon[32944]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:30:36.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:35 smithi142 ceph-mon[32944]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T21:30:36.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:35 smithi050 ceph-mon[32753]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:30:36.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:35 smithi050 ceph-mon[32753]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 287 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:30:36.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:35 smithi050 ceph-mon[32753]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T21:30:38.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:37 smithi142 ceph-mon[32944]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-26T21:30:38.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:37 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/2699607002' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e55570e-77e5-4892-8b9f-77dcd03a4eec"}]: dispatch 2024-04-26T21:30:38.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:37 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/2699607002' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e55570e-77e5-4892-8b9f-77dcd03a4eec"}]': finished 2024-04-26T21:30:38.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:37 smithi142 ceph-mon[32944]: osdmap e64: 10 total, 9 up, 10 in 2024-04-26T21:30:38.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:37 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:30:38.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:37 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/2710139676' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:30:38.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:37 smithi046 ceph-mon[29201]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-26T21:30:38.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:37 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2699607002' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e55570e-77e5-4892-8b9f-77dcd03a4eec"}]: dispatch 2024-04-26T21:30:38.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:37 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2699607002' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e55570e-77e5-4892-8b9f-77dcd03a4eec"}]': finished 2024-04-26T21:30:38.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:37 smithi046 ceph-mon[29201]: osdmap e64: 10 total, 9 up, 10 in 2024-04-26T21:30:38.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:37 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:30:38.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:37 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2710139676' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:30:38.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:37 smithi050 ceph-mon[32753]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-26T21:30:38.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:37 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/2699607002' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e55570e-77e5-4892-8b9f-77dcd03a4eec"}]: dispatch 2024-04-26T21:30:38.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:37 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/2699607002' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e55570e-77e5-4892-8b9f-77dcd03a4eec"}]': finished 2024-04-26T21:30:38.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:37 smithi050 ceph-mon[32753]: osdmap e64: 10 total, 9 up, 10 in 2024-04-26T21:30:38.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:37 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:30:38.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:37 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/2710139676' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:30:40.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:39 smithi142 ceph-mon[32944]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-26T21:30:40.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:39 smithi046 ceph-mon[29201]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-26T21:30:40.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:39 smithi050 ceph-mon[32753]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-04-26T21:30:41.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:41 smithi046 ceph-mon[29201]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-04-26T21:30:41.920 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:41 smithi050 ceph-mon[32753]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-04-26T21:30:42.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:41 smithi142 ceph-mon[32944]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-04-26T21:30:43.793 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:43 smithi046 ceph-mon[29201]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T21:30:44.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:43 smithi142 ceph-mon[32944]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T21:30:44.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:43 smithi050 ceph-mon[32753]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T21:30:45.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:45 smithi046 ceph-mon[29201]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-26T21:30:45.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:45 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T21:30:45.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:45 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:45.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:45 smithi046 ceph-mon[29201]: Deploying daemon osd.9 on smithi046 2024-04-26T21:30:46.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:45 smithi142 ceph-mon[32944]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-26T21:30:46.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:45 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T21:30:46.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:45 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:46.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:45 smithi142 ceph-mon[32944]: Deploying daemon osd.9 on smithi046 2024-04-26T21:30:46.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:45 smithi050 ceph-mon[32753]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-26T21:30:46.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:45 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T21:30:46.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:45 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:46.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:45 smithi050 ceph-mon[32753]: Deploying daemon osd.9 on smithi046 2024-04-26T21:30:48.021 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:47 smithi046 ceph-mon[29201]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:30:48.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:47 smithi142 ceph-mon[32944]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:30:48.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:47 smithi050 ceph-mon[32753]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:30:49.913 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:49 smithi046 ceph-mon[29201]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:30:50.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:49 smithi142 ceph-mon[32944]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:30:50.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:49 smithi050 ceph-mon[32753]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:30:51.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:51 smithi046 ceph-mon[29201]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:30:51.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:51 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:51.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:51 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:51.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:51 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:30:52.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:51 smithi142 ceph-mon[32944]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:30:52.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:51 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:52.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:51 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:52.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:51 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:30:52.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:51 smithi050 ceph-mon[32753]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:30:52.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:51 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:52.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:51 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:52.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:51 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:30:53.744 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:53 smithi046 ceph-mon[29201]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:30:54.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:53 smithi142 ceph-mon[32944]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:30:54.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:53 smithi050 ceph-mon[32753]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:30:55.112 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:55 smithi046 ceph-mon[29201]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:30:55.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:55 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:55.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:55 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:55.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:55 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:55.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:55 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:30:55.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:55 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:55.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:55 smithi046 ceph-mon[29201]: from='osd.9 [v2:172.21.15.46:6826/3601947736,v1:172.21.15.46:6827/3601947736]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T21:30:55.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:55.132+0000 7f42dd7fa700 1 -- 172.21.15.46:0/13601090 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f42e0063860 con 0x7f42c80a3c40 2024-04-26T21:30:55.134 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 9 on host 'smithi046' 2024-04-26T21:30:55.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:55.135+0000 7f42e6a54700 1 -- 172.21.15.46:0/13601090 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f42c80a3c40 msgr2=0x7f42c80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:55.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:55.135+0000 7f42e6a54700 1 --2- 172.21.15.46:0/13601090 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f42c80a3c40 0x7f42c80a6100 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f42d0009c60 tx=0x7f42d0009260 comp rx=0 tx=0).stop 2024-04-26T21:30:55.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:55.135+0000 7f42e6a54700 1 -- 172.21.15.46:0/13601090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42e01068a0 msgr2=0x7f42e0199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:55.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:55.135+0000 7f42e6a54700 1 --2- 172.21.15.46:0/13601090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42e01068a0 0x7f42e0199b10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f42d400eb60 tx=0x7f42d400ee70 comp rx=0 tx=0).stop 2024-04-26T21:30:55.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:55.135+0000 7f42e6a54700 1 -- 172.21.15.46:0/13601090 shutdown_connections 2024-04-26T21:30:55.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:55.135+0000 7f42e6a54700 1 --2- 172.21.15.46:0/13601090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f42e01091d0 0x7f42e01a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:55.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:55.135+0000 7f42e6a54700 1 --2- 172.21.15.46:0/13601090 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f42e0103f70 0x7f42e01995d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:55.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:55.135+0000 7f42e6a54700 1 --2- 172.21.15.46:0/13601090 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f42c80a3c40 0x7f42c80a6100 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:55.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:55.135+0000 7f42e6a54700 1 --2- 172.21.15.46:0/13601090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42e01068a0 0x7f42e0199b10 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:55.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:55.135+0000 7f42e6a54700 1 -- 172.21.15.46:0/13601090 >> 172.21.15.46:0/13601090 conn(0x7f42e00fd920 msgr2=0x7f42e0104810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:55.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:55.135+0000 7f42e6a54700 1 -- 172.21.15.46:0/13601090 shutdown_connections 2024-04-26T21:30:55.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:30:55.135+0000 7f42e6a54700 1 -- 172.21.15.46:0/13601090 wait complete. 2024-04-26T21:30:55.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:55 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:55.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:55 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:30:55.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:55 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:55.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:55 smithi050 ceph-mon[32753]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:30:55.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:55 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:55.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:55 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:55.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:55 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:55.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:55 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:30:55.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:55 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:55.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:55 smithi050 ceph-mon[32753]: from='osd.9 [v2:172.21.15.46:6826/3601947736,v1:172.21.15.46:6827/3601947736]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T21:30:55.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:55 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:55.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:55 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:30:55.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:55 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:55.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:55 smithi142 ceph-mon[32944]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:30:55.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:55 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:55.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:55 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:55.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:55 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:55.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:55 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:30:55.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:55 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:55.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:55 smithi142 ceph-mon[32944]: from='osd.9 [v2:172.21.15.46:6826/3601947736,v1:172.21.15.46:6827/3601947736]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T21:30:55.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:55 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:55.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:55 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:30:55.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:55 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:30:55.989 DEBUG:teuthology.orchestra.run.smithi046:osd.9> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.9.service 2024-04-26T21:30:55.991 INFO:tasks.cephadm:Deploying osd.10 on smithi050 with /dev/vg_nvme/lv_1... 2024-04-26T21:30:55.991 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T21:30:56.256 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T21:30:57.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:56 smithi142 ceph-mon[32944]: from='osd.9 [v2:172.21.15.46:6826/3601947736,v1:172.21.15.46:6827/3601947736]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T21:30:57.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:56 smithi142 ceph-mon[32944]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T21:30:57.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:56 smithi142 ceph-mon[32944]: from='osd.9 [v2:172.21.15.46:6826/3601947736,v1:172.21.15.46:6827/3601947736]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-26T21:30:57.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:56 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:30:57.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:56 smithi046 ceph-mon[29201]: from='osd.9 [v2:172.21.15.46:6826/3601947736,v1:172.21.15.46:6827/3601947736]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T21:30:57.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:56 smithi046 ceph-mon[29201]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T21:30:57.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:56 smithi046 ceph-mon[29201]: from='osd.9 [v2:172.21.15.46:6826/3601947736,v1:172.21.15.46:6827/3601947736]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-26T21:30:57.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:56 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:30:57.054 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:30:56 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:30:56.683+0000 7f25dcce5700 -1 osd.9 0 waiting for initial osdmap 2024-04-26T21:30:57.054 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:30:56 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:30:56.694+0000 7f25d8ad8700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T21:30:57.075 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:56 smithi050 ceph-mon[32753]: from='osd.9 [v2:172.21.15.46:6826/3601947736,v1:172.21.15.46:6827/3601947736]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T21:30:57.075 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:56 smithi050 ceph-mon[32753]: osdmap e65: 10 total, 9 up, 10 in 2024-04-26T21:30:57.075 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:56 smithi050 ceph-mon[32753]: from='osd.9 [v2:172.21.15.46:6826/3601947736,v1:172.21.15.46:6827/3601947736]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-04-26T21:30:57.075 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:56 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:30:57.801 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:57 smithi050 ceph-mon[32753]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:30:57.801 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:57 smithi050 ceph-mon[32753]: from='osd.9 [v2:172.21.15.46:6826/3601947736,v1:172.21.15.46:6827/3601947736]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-26T21:30:57.801 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:57 smithi050 ceph-mon[32753]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T21:30:57.801 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:57 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:30:57.801 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:57 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:30:58.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:57 smithi142 ceph-mon[32944]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:30:58.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:57 smithi142 ceph-mon[32944]: from='osd.9 [v2:172.21.15.46:6826/3601947736,v1:172.21.15.46:6827/3601947736]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-26T21:30:58.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:57 smithi142 ceph-mon[32944]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T21:30:58.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:57 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:30:58.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:57 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:30:58.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:57 smithi046 ceph-mon[29201]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:30:58.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:57 smithi046 ceph-mon[29201]: from='osd.9 [v2:172.21.15.46:6826/3601947736,v1:172.21.15.46:6827/3601947736]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-04-26T21:30:58.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:57 smithi046 ceph-mon[29201]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T21:30:58.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:57 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:30:58.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:57 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:30:58.399 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T21:30:58.399 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T21:30:58.399 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10+0 records in 2024-04-26T21:30:58.399 INFO:teuthology.orchestra.run.smithi050.stderr:10+0 records out 2024-04-26T21:30:58.400 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00846036 s, 1.2 GB/s 2024-04-26T21:30:58.400 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping successful for: 2024-04-26T21:30:59.018 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:58 smithi050 ceph-mon[32753]: purged_snaps scrub starts 2024-04-26T21:30:59.018 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:58 smithi050 ceph-mon[32753]: purged_snaps scrub ok 2024-04-26T21:30:59.018 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:58 smithi050 ceph-mon[32753]: osd.9 [v2:172.21.15.46:6826/3601947736,v1:172.21.15.46:6827/3601947736] boot 2024-04-26T21:30:59.019 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:58 smithi050 ceph-mon[32753]: osdmap e67: 10 total, 10 up, 10 in 2024-04-26T21:30:59.019 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:58 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:30:59.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:58 smithi142 ceph-mon[32944]: purged_snaps scrub starts 2024-04-26T21:30:59.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:58 smithi142 ceph-mon[32944]: purged_snaps scrub ok 2024-04-26T21:30:59.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:58 smithi142 ceph-mon[32944]: osd.9 [v2:172.21.15.46:6826/3601947736,v1:172.21.15.46:6827/3601947736] boot 2024-04-26T21:30:59.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:58 smithi142 ceph-mon[32944]: osdmap e67: 10 total, 10 up, 10 in 2024-04-26T21:30:59.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:58 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:30:59.046 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi050:vg_nvme/lv_1 2024-04-26T21:30:59.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:58 smithi046 ceph-mon[29201]: purged_snaps scrub starts 2024-04-26T21:30:59.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:58 smithi046 ceph-mon[29201]: purged_snaps scrub ok 2024-04-26T21:30:59.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:58 smithi046 ceph-mon[29201]: osd.9 [v2:172.21.15.46:6826/3601947736,v1:172.21.15.46:6827/3601947736] boot 2024-04-26T21:30:59.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:58 smithi046 ceph-mon[29201]: osdmap e67: 10 total, 10 up, 10 in 2024-04-26T21:30:59.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:58 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:30:59.305 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T21:31:00.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:59 smithi142 ceph-mon[32944]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:00.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:59 smithi142 ceph-mon[32944]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T21:31:00.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:59 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:00.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:59 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:00.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:59 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:00.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:59 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:00.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:59 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:00.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:59 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:00.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:59 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:00.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:59 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:00.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:59 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:31:00.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:30:59 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:00.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:59 smithi046 ceph-mon[29201]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:00.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:59 smithi046 ceph-mon[29201]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T21:31:00.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:59 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:00.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:59 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:00.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:59 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:00.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:59 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:00.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:59 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:00.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:59 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:00.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:59 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:00.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:59 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:00.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:59 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:31:00.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:30:59 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:00.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:59 smithi050 ceph-mon[32753]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:00.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:59 smithi050 ceph-mon[32753]: osdmap e68: 10 total, 10 up, 10 in 2024-04-26T21:31:00.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:59 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:00.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:59 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:00.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:59 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:00.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:59 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:00.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:59 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:00.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:59 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:00.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:59 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:00.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:59 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:00.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:59 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:31:00.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:30:59 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:00.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.978+0000 7f7860fa1700 1 -- 172.21.15.50:0/1053806315 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f785c103f40 msgr2=0x7f785c10a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:00.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.978+0000 7f7860fa1700 1 --2- 172.21.15.50:0/1053806315 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f785c103f40 0x7f785c10a7f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f785400b0d0 tx=0x7f785400b3e0 comp rx=0 tx=0).stop 2024-04-26T21:31:00.980 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:00 smithi050 ceph-mon[32753]: Detected new or changed devices on smithi046 2024-04-26T21:31:00.980 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:00 smithi050 ceph-mon[32753]: Adjusting osd_memory_target on smithi046 to 4273M 2024-04-26T21:31:00.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.978+0000 7f7860fa1700 1 -- 172.21.15.50:0/1053806315 shutdown_connections 2024-04-26T21:31:00.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.978+0000 7f7860fa1700 1 --2- 172.21.15.50:0/1053806315 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f785c103f40 0x7f785c10a7f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:00.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.978+0000 7f7860fa1700 1 --2- 172.21.15.50:0/1053806315 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f785c1023a0 0x7f785c1027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:00.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.978+0000 7f7860fa1700 1 --2- 172.21.15.50:0/1053806315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c1035a0 0x7f785c103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:00.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.978+0000 7f7860fa1700 1 -- 172.21.15.50:0/1053806315 >> 172.21.15.50:0/1053806315 conn(0x7f785c0fd940 msgr2=0x7f785c0ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:00.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.979+0000 7f7860fa1700 1 -- 172.21.15.50:0/1053806315 shutdown_connections 2024-04-26T21:31:00.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.979+0000 7f7860fa1700 1 -- 172.21.15.50:0/1053806315 wait complete. 2024-04-26T21:31:00.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.979+0000 7f7860fa1700 1 Processor -- start 2024-04-26T21:31:00.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.979+0000 7f7860fa1700 1 -- start start 2024-04-26T21:31:00.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.980+0000 7f7860fa1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f785c1023a0 0x7f785c06c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:00.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.980+0000 7f7860fa1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f785c1035a0 0x7f785c1a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:00.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.980+0000 7f7860fa1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c103f40 0x7f785c1a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:00.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.980+0000 7f7860fa1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f785c10deb0 con 0x7f785c103f40 2024-04-26T21:31:00.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.980+0000 7f7860fa1700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f785c10dd30 con 0x7f785c1023a0 2024-04-26T21:31:00.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.980+0000 7f7860fa1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f785c10e030 con 0x7f785c1035a0 2024-04-26T21:31:00.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.980+0000 7f785a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f785c1023a0 0x7f785c06c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:00.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.980+0000 7f785a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f785c1023a0 0x7f785c06c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47966/0 (socket says 172.21.15.50:47966) 2024-04-26T21:31:00.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.980+0000 7f785a59c700 1 -- 172.21.15.50:0/1284756084 learned_addr learned my addr 172.21.15.50:0/1284756084 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T21:31:00.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.980+0000 7f7859d9b700 1 --2- 172.21.15.50:0/1284756084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f785c1035a0 0x7f785c1a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:00.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.980+0000 7f785ad9d700 1 --2- 172.21.15.50:0/1284756084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c103f40 0x7f785c1a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:00.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.980+0000 7f785a59c700 1 -- 172.21.15.50:0/1284756084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f785c1035a0 msgr2=0x7f785c1a06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:00.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.980+0000 7f785a59c700 1 --2- 172.21.15.50:0/1284756084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f785c1035a0 0x7f785c1a06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:00.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.980+0000 7f785a59c700 1 -- 172.21.15.50:0/1284756084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c103f40 msgr2=0x7f785c1a2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:00.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.980+0000 7f785a59c700 1 --2- 172.21.15.50:0/1284756084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c103f40 0x7f785c1a2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:00.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.980+0000 7f785a59c700 1 -- 172.21.15.50:0/1284756084 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f785c1a3420 con 0x7f785c1023a0 2024-04-26T21:31:00.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.980+0000 7f785a59c700 1 --2- 172.21.15.50:0/1284756084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f785c1023a0 0x7f785c06c6b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f784c00b500 tx=0x7f784c00b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:00.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.981+0000 7f78537fe700 1 -- 172.21.15.50:0/1284756084 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f784c015020 con 0x7f785c1023a0 2024-04-26T21:31:00.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.981+0000 7f7860fa1700 1 -- 172.21.15.50:0/1284756084 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f785c1a3710 con 0x7f785c1023a0 2024-04-26T21:31:00.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.981+0000 7f78537fe700 1 -- 172.21.15.50:0/1284756084 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f784c011ce0 con 0x7f785c1023a0 2024-04-26T21:31:00.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.981+0000 7f78537fe700 1 -- 172.21.15.50:0/1284756084 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f784c010c80 con 0x7f785c1023a0 2024-04-26T21:31:00.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.981+0000 7f7860fa1700 1 -- 172.21.15.50:0/1284756084 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f785c1a3d20 con 0x7f785c1023a0 2024-04-26T21:31:00.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.982+0000 7f7860fa1700 1 -- 172.21.15.50:0/1284756084 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f785c06a110 con 0x7f785c1023a0 2024-04-26T21:31:00.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.983+0000 7f78537fe700 1 -- 172.21.15.50:0/1284756084 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f784c018020 con 0x7f785c1023a0 2024-04-26T21:31:00.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.984+0000 7f78537fe700 1 --2- 172.21.15.50:0/1284756084 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f78480a3c40 0x7f78480a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:00.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.984+0000 7f78537fe700 1 -- 172.21.15.50:0/1284756084 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f784c0cf890 con 0x7f785c1023a0 2024-04-26T21:31:00.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.984+0000 7f7859d9b700 1 --2- 172.21.15.50:0/1284756084 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f78480a3c40 0x7f78480a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:00.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.984+0000 7f7859d9b700 1 --2- 172.21.15.50:0/1284756084 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f78480a3c40 0x7f78480a6100 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f785c06cc30 tx=0x7f784400a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:00.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:00.987+0000 7f78537fe700 1 -- 172.21.15.50:0/1284756084 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f784c01f020 con 0x7f785c1023a0 2024-04-26T21:31:01.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:00 smithi142 ceph-mon[32944]: Detected new or changed devices on smithi046 2024-04-26T21:31:01.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:00 smithi142 ceph-mon[32944]: Adjusting osd_memory_target on smithi046 to 4273M 2024-04-26T21:31:01.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:00 smithi046 ceph-mon[29201]: Detected new or changed devices on smithi046 2024-04-26T21:31:01.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:00 smithi046 ceph-mon[29201]: Adjusting osd_memory_target on smithi046 to 4273M 2024-04-26T21:31:01.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:01.205+0000 7f7860fa1700 1 -- 172.21.15.50:0/1284756084 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f785c063860 con 0x7f78480a3c40 2024-04-26T21:31:01.966 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:01 smithi050 ceph-mon[32753]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:01.966 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:01 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:31:01.966 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:01 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:31:01.966 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:01 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:02.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:01 smithi142 ceph-mon[32944]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:02.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:01 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:31:02.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:01 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:31:02.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:01 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:02.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:01 smithi046 ceph-mon[29201]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:02.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:01 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:31:02.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:01 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:31:02.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:01 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:03.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:02 smithi142 ceph-mon[32944]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:31:03.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:02 smithi046 ceph-mon[29201]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:31:03.057 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:02 smithi050 ceph-mon[32753]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:31:04.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:03 smithi142 ceph-mon[32944]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:04.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:03 smithi046 ceph-mon[29201]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:04.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:03 smithi050 ceph-mon[32753]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:05.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:04 smithi142 ceph-mon[32944]: from='client.? 172.21.15.50:0/1217152826' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "366389fa-fb9a-408b-b5c2-343c054c5a64"}]: dispatch 2024-04-26T21:31:05.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:04 smithi142 ceph-mon[32944]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "366389fa-fb9a-408b-b5c2-343c054c5a64"}]: dispatch 2024-04-26T21:31:05.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:04 smithi142 ceph-mon[32944]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "366389fa-fb9a-408b-b5c2-343c054c5a64"}]': finished 2024-04-26T21:31:05.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:04 smithi142 ceph-mon[32944]: osdmap e69: 11 total, 10 up, 11 in 2024-04-26T21:31:05.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:04 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:31:05.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:04 smithi142 ceph-mon[32944]: from='client.? 172.21.15.50:0/603657933' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:31:05.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:04 smithi046 ceph-mon[29201]: from='client.? 172.21.15.50:0/1217152826' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "366389fa-fb9a-408b-b5c2-343c054c5a64"}]: dispatch 2024-04-26T21:31:05.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:04 smithi046 ceph-mon[29201]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "366389fa-fb9a-408b-b5c2-343c054c5a64"}]: dispatch 2024-04-26T21:31:05.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:04 smithi046 ceph-mon[29201]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "366389fa-fb9a-408b-b5c2-343c054c5a64"}]': finished 2024-04-26T21:31:05.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:04 smithi046 ceph-mon[29201]: osdmap e69: 11 total, 10 up, 11 in 2024-04-26T21:31:05.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:04 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:31:05.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:04 smithi046 ceph-mon[29201]: from='client.? 172.21.15.50:0/603657933' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:31:05.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:04 smithi050 ceph-mon[32753]: from='client.? 172.21.15.50:0/1217152826' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "366389fa-fb9a-408b-b5c2-343c054c5a64"}]: dispatch 2024-04-26T21:31:05.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:04 smithi050 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "366389fa-fb9a-408b-b5c2-343c054c5a64"}]: dispatch 2024-04-26T21:31:05.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:04 smithi050 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "366389fa-fb9a-408b-b5c2-343c054c5a64"}]': finished 2024-04-26T21:31:05.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:04 smithi050 ceph-mon[32753]: osdmap e69: 11 total, 10 up, 11 in 2024-04-26T21:31:05.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:04 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:31:05.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:04 smithi050 ceph-mon[32753]: from='client.? 172.21.15.50:0/603657933' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:31:06.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:05 smithi142 ceph-mon[32944]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:06.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:05 smithi046 ceph-mon[29201]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:06.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:05 smithi050 ceph-mon[32753]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:08.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:07 smithi142 ceph-mon[32944]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:08.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:07 smithi046 ceph-mon[29201]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:08.114 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:07 smithi050 ceph-mon[32753]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:10.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:09 smithi142 ceph-mon[32944]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:10.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:09 smithi046 ceph-mon[29201]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:10.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:09 smithi050 ceph-mon[32753]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:12.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:11 smithi142 ceph-mon[32944]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:12.070 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:11 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T21:31:12.071 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:11 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:12.071 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:11 smithi046 ceph-mon[29201]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:12.071 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:11 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T21:31:12.071 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:11 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:12.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:11 smithi050 ceph-mon[32753]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:12.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:11 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T21:31:12.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:11 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:13.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:12 smithi142 ceph-mon[32944]: Deploying daemon osd.10 on smithi050 2024-04-26T21:31:13.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:12 smithi046 ceph-mon[29201]: Deploying daemon osd.10 on smithi050 2024-04-26T21:31:13.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:12 smithi050 ceph-mon[32753]: Deploying daemon osd.10 on smithi050 2024-04-26T21:31:14.036 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:13 smithi050 ceph-mon[32753]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:14.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:13 smithi142 ceph-mon[32944]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:14.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:13 smithi046 ceph-mon[29201]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:15.831 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:15 smithi050 ceph-mon[32753]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:16.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:15 smithi142 ceph-mon[32944]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:16.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:15 smithi046 ceph-mon[29201]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:18.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:17 smithi142 ceph-mon[32944]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:18.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:17 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:18.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:17 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:18.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:17 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:31:18.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:17 smithi046 ceph-mon[29201]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:18.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:17 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:18.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:17 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:18.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:17 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:31:18.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:17 smithi050 ceph-mon[32753]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:18.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:17 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:18.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:17 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:18.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:17 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:31:20.002 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:19 smithi050 ceph-mon[32753]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:20.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:19 smithi142 ceph-mon[32944]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:20.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:19 smithi046 ceph-mon[29201]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:20.827+0000 7f78537fe700 1 -- 172.21.15.50:0/1284756084 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f785c063860 con 0x7f78480a3c40 2024-04-26T21:31:20.828 INFO:teuthology.orchestra.run.smithi050.stdout:Created osd(s) 10 on host 'smithi050' 2024-04-26T21:31:20.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:20.829+0000 7f7860fa1700 1 -- 172.21.15.50:0/1284756084 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f78480a3c40 msgr2=0x7f78480a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:20.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:20.829+0000 7f7860fa1700 1 --2- 172.21.15.50:0/1284756084 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f78480a3c40 0x7f78480a6100 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f785c06cc30 tx=0x7f784400a300 comp rx=0 tx=0).stop 2024-04-26T21:31:20.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:20.829+0000 7f7860fa1700 1 -- 172.21.15.50:0/1284756084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f785c1023a0 msgr2=0x7f785c06c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:20.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:20.829+0000 7f7860fa1700 1 --2- 172.21.15.50:0/1284756084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f785c1023a0 0x7f785c06c6b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f784c00b500 tx=0x7f784c00b810 comp rx=0 tx=0).stop 2024-04-26T21:31:20.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:20.829+0000 7f7860fa1700 1 -- 172.21.15.50:0/1284756084 shutdown_connections 2024-04-26T21:31:20.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:20.829+0000 7f7860fa1700 1 --2- 172.21.15.50:0/1284756084 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f78480a3c40 0x7f78480a6100 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:20.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:20.829+0000 7f7860fa1700 1 --2- 172.21.15.50:0/1284756084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f785c103f40 0x7f785c1a2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:20.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:20.829+0000 7f7860fa1700 1 --2- 172.21.15.50:0/1284756084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f785c1023a0 0x7f785c06c6b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:20.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:20.829+0000 7f7860fa1700 1 --2- 172.21.15.50:0/1284756084 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f785c1035a0 0x7f785c1a06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:20.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:20.829+0000 7f7860fa1700 1 -- 172.21.15.50:0/1284756084 >> 172.21.15.50:0/1284756084 conn(0x7f785c0fd940 msgr2=0x7f785c1068c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:20.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:20.829+0000 7f7860fa1700 1 -- 172.21.15.50:0/1284756084 shutdown_connections 2024-04-26T21:31:20.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T21:31:20.829+0000 7f7860fa1700 1 -- 172.21.15.50:0/1284756084 wait complete. 2024-04-26T21:31:21.682 DEBUG:teuthology.orchestra.run.smithi050:osd.10> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.10.service 2024-04-26T21:31:21.685 INFO:tasks.cephadm:Deploying osd.11 on smithi142 with /dev/vg_nvme/lv_1... 2024-04-26T21:31:21.685 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T21:31:21.937 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:31:22.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:21 smithi142 ceph-mon[32944]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:22.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:21 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:22.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:21 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:22.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:21 smithi142 ceph-mon[32944]: from='osd.10 [v2:172.21.15.50:6824/3135765006,v1:172.21.15.50:6825/3135765006]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T21:31:22.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:21 smithi142 ceph-mon[32944]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T21:31:22.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:21 smithi046 ceph-mon[29201]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:22.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:21 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:22.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:21 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:22.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:21 smithi046 ceph-mon[29201]: from='osd.10 [v2:172.21.15.50:6824/3135765006,v1:172.21.15.50:6825/3135765006]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T21:31:22.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:21 smithi046 ceph-mon[29201]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T21:31:22.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:21 smithi050 ceph-mon[32753]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:22.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:21 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:22.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:21 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:22.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:21 smithi050 ceph-mon[32753]: from='osd.10 [v2:172.21.15.50:6824/3135765006,v1:172.21.15.50:6825/3135765006]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T21:31:22.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:21 smithi050 ceph-mon[32753]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T21:31:22.811 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:22 smithi050 ceph-mon[32753]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T21:31:22.811 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:22 smithi050 ceph-mon[32753]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T21:31:22.811 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:22 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:31:22.811 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:22 smithi050 ceph-mon[32753]: from='osd.10 [v2:172.21.15.50:6824/3135765006,v1:172.21.15.50:6825/3135765006]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T21:31:22.811 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:22 smithi050 ceph-mon[32753]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T21:31:22.811 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:22 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:22.811 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:22 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:23.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:22 smithi046 ceph-mon[29201]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T21:31:23.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:22 smithi046 ceph-mon[29201]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T21:31:23.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:22 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:31:23.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:22 smithi046 ceph-mon[29201]: from='osd.10 [v2:172.21.15.50:6824/3135765006,v1:172.21.15.50:6825/3135765006]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T21:31:23.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:22 smithi046 ceph-mon[29201]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T21:31:23.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:22 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:23.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:22 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:23.062 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:22 smithi142 ceph-mon[32944]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T21:31:23.062 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:22 smithi142 ceph-mon[32944]: osdmap e70: 11 total, 10 up, 11 in 2024-04-26T21:31:23.062 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:22 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:31:23.062 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:22 smithi142 ceph-mon[32944]: from='osd.10 [v2:172.21.15.50:6824/3135765006,v1:172.21.15.50:6825/3135765006]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T21:31:23.062 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:22 smithi142 ceph-mon[32944]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T21:31:23.062 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:22 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:23.063 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:22 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:23.567 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:31:23 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:31:23.278+0000 7fae98a0c700 -1 osd.10 0 waiting for initial osdmap 2024-04-26T21:31:23.568 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:31:23 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:31:23.289+0000 7fae907f7700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T21:31:23.840 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:23 smithi050 ceph-mon[32753]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:23.840 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:23 smithi050 ceph-mon[32753]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T21:31:23.840 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:23 smithi050 ceph-mon[32753]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T21:31:23.840 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:23 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:31:23.841 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:23 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:31:23.841 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:23 smithi050 ceph-mon[32753]: from='osd.10 ' entity='osd.10' 2024-04-26T21:31:23.979 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T21:31:23.979 INFO:teuthology.orchestra.run.smithi142.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T21:31:23.979 INFO:teuthology.orchestra.run.smithi142.stderr: stderr: 10+0 records in 2024-04-26T21:31:23.979 INFO:teuthology.orchestra.run.smithi142.stderr:10+0 records out 2024-04-26T21:31:23.979 INFO:teuthology.orchestra.run.smithi142.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112772 s, 930 MB/s 2024-04-26T21:31:23.979 INFO:teuthology.orchestra.run.smithi142.stderr:--> Zapping successful for: 2024-04-26T21:31:24.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:23 smithi046 ceph-mon[29201]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:24.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:23 smithi046 ceph-mon[29201]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T21:31:24.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:23 smithi046 ceph-mon[29201]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T21:31:24.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:23 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:31:24.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:23 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:31:24.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:23 smithi046 ceph-mon[29201]: from='osd.10 ' entity='osd.10' 2024-04-26T21:31:24.279 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:23 smithi142 ceph-mon[32944]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:24.279 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:23 smithi142 ceph-mon[32944]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T21:31:24.279 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:23 smithi142 ceph-mon[32944]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T21:31:24.279 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:23 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:31:24.279 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:23 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:31:24.279 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:23 smithi142 ceph-mon[32944]: from='osd.10 ' entity='osd.10' 2024-04-26T21:31:24.727 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi142:vg_nvme/lv_1 2024-04-26T21:31:24.975 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:31:25.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:24 smithi142 ceph-mon[32944]: purged_snaps scrub starts 2024-04-26T21:31:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:24 smithi142 ceph-mon[32944]: purged_snaps scrub ok 2024-04-26T21:31:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:24 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:31:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:24 smithi142 ceph-mon[32944]: osd.10 [v2:172.21.15.50:6824/3135765006,v1:172.21.15.50:6825/3135765006] boot 2024-04-26T21:31:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:24 smithi142 ceph-mon[32944]: osdmap e72: 11 total, 11 up, 11 in 2024-04-26T21:31:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:24 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:31:25.072 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:24 smithi050 ceph-mon[32753]: purged_snaps scrub starts 2024-04-26T21:31:25.072 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:24 smithi050 ceph-mon[32753]: purged_snaps scrub ok 2024-04-26T21:31:25.072 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:24 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:31:25.073 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:24 smithi050 ceph-mon[32753]: osd.10 [v2:172.21.15.50:6824/3135765006,v1:172.21.15.50:6825/3135765006] boot 2024-04-26T21:31:25.073 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:24 smithi050 ceph-mon[32753]: osdmap e72: 11 total, 11 up, 11 in 2024-04-26T21:31:25.073 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:24 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:31:25.207 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:24 smithi046 ceph-mon[29201]: purged_snaps scrub starts 2024-04-26T21:31:25.207 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:24 smithi046 ceph-mon[29201]: purged_snaps scrub ok 2024-04-26T21:31:25.207 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:24 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:31:25.207 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:24 smithi046 ceph-mon[29201]: osd.10 [v2:172.21.15.50:6824/3135765006,v1:172.21.15.50:6825/3135765006] boot 2024-04-26T21:31:25.207 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:24 smithi046 ceph-mon[29201]: osdmap e72: 11 total, 11 up, 11 in 2024-04-26T21:31:25.207 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:24 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:31:26.098 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:25 smithi142 ceph-mon[32944]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:26.098 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:25 smithi142 ceph-mon[32944]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T21:31:26.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:25 smithi050 ceph-mon[32753]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:26.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:25 smithi050 ceph-mon[32753]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T21:31:26.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:25 smithi046 ceph-mon[29201]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:31:26.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:25 smithi046 ceph-mon[29201]: osdmap e73: 11 total, 11 up, 11 in 2024-04-26T21:31:26.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.561+0000 7fbb72ab7700 1 -- 172.21.15.142:0/1572564103 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb6c103490 msgr2=0x7fbb6c1038b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:26.564 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.561+0000 7fbb72ab7700 1 --2- 172.21.15.142:0/1572564103 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb6c103490 0x7fbb6c1038b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbb60009990 tx=0x7fbb60009ca0 comp rx=0 tx=0).stop 2024-04-26T21:31:26.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.561+0000 7fbb72ab7700 1 -- 172.21.15.142:0/1572564103 shutdown_connections 2024-04-26T21:31:26.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.561+0000 7fbb72ab7700 1 --2- 172.21.15.142:0/1572564103 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb6c105030 0x7fbb6c10b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:26.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.561+0000 7fbb72ab7700 1 --2- 172.21.15.142:0/1572564103 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb6c103490 0x7fbb6c1038b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:26.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.561+0000 7fbb72ab7700 1 --2- 172.21.15.142:0/1572564103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb6c104690 0x7fbb6c104af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:26.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.561+0000 7fbb72ab7700 1 -- 172.21.15.142:0/1572564103 >> 172.21.15.142:0/1572564103 conn(0x7fbb6c0fea30 msgr2=0x7fbb6c100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:26.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.562+0000 7fbb72ab7700 1 -- 172.21.15.142:0/1572564103 shutdown_connections 2024-04-26T21:31:26.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.562+0000 7fbb72ab7700 1 -- 172.21.15.142:0/1572564103 wait complete. 2024-04-26T21:31:26.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.562+0000 7fbb72ab7700 1 Processor -- start 2024-04-26T21:31:26.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.562+0000 7fbb72ab7700 1 -- start start 2024-04-26T21:31:26.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.562+0000 7fbb72ab7700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb6c103490 0x7fbb6c0771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:26.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.562+0000 7fbb72ab7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb6c104690 0x7fbb6c075810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:26.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.562+0000 7fbb72ab7700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb6c105030 0x7fbb6c075d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:26.565 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.562+0000 7fbb72ab7700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbb6c10ef10 con 0x7fbb6c105030 2024-04-26T21:31:26.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.562+0000 7fbb72ab7700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbb6c10ed90 con 0x7fbb6c104690 2024-04-26T21:31:26.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.562+0000 7fbb72ab7700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbb6c10f090 con 0x7fbb6c103490 2024-04-26T21:31:26.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.562+0000 7fbb70853700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb6c103490 0x7fbb6c0771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:26.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.562+0000 7fbb70853700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb6c103490 0x7fbb6c0771c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:39320/0 (socket says 172.21.15.142:39320) 2024-04-26T21:31:26.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.562+0000 7fbb70853700 1 -- 172.21.15.142:0/2070288261 learned_addr learned my addr 172.21.15.142:0/2070288261 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-04-26T21:31:26.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.562+0000 7fbb71054700 1 --2- 172.21.15.142:0/2070288261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb6c105030 0x7fbb6c075d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:26.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.563+0000 7fbb6bfff700 1 --2- 172.21.15.142:0/2070288261 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb6c104690 0x7fbb6c075810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:26.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.563+0000 7fbb70853700 1 -- 172.21.15.142:0/2070288261 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb6c104690 msgr2=0x7fbb6c075810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:26.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.563+0000 7fbb70853700 1 --2- 172.21.15.142:0/2070288261 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb6c104690 0x7fbb6c075810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:26.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.563+0000 7fbb70853700 1 -- 172.21.15.142:0/2070288261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb6c105030 msgr2=0x7fbb6c075d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:26.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.563+0000 7fbb70853700 1 --2- 172.21.15.142:0/2070288261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb6c105030 0x7fbb6c075d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:26.566 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.563+0000 7fbb70853700 1 -- 172.21.15.142:0/2070288261 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb6c076600 con 0x7fbb6c103490 2024-04-26T21:31:26.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.563+0000 7fbb6bfff700 1 --2- 172.21.15.142:0/2070288261 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb6c104690 0x7fbb6c075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:31:26.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.563+0000 7fbb70853700 1 --2- 172.21.15.142:0/2070288261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb6c103490 0x7fbb6c0771c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbb600036f0 tx=0x7fbb60009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:26.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.563+0000 7fbb69ffb700 1 -- 172.21.15.142:0/2070288261 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb60027070 con 0x7fbb6c103490 2024-04-26T21:31:26.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.563+0000 7fbb69ffb700 1 -- 172.21.15.142:0/2070288261 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb6000bd90 con 0x7fbb6c103490 2024-04-26T21:31:26.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.563+0000 7fbb72ab7700 1 -- 172.21.15.142:0/2070288261 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb6c1a3680 con 0x7fbb6c103490 2024-04-26T21:31:26.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.563+0000 7fbb69ffb700 1 -- 172.21.15.142:0/2070288261 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb60022db0 con 0x7fbb6c103490 2024-04-26T21:31:26.567 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.563+0000 7fbb72ab7700 1 -- 172.21.15.142:0/2070288261 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb6c1a3ad0 con 0x7fbb6c103490 2024-04-26T21:31:26.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.564+0000 7fbb72ab7700 1 -- 172.21.15.142:0/2070288261 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb6c06a110 con 0x7fbb6c103490 2024-04-26T21:31:26.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.565+0000 7fbb69ffb700 1 -- 172.21.15.142:0/2070288261 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbb6002a070 con 0x7fbb6c103490 2024-04-26T21:31:26.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.566+0000 7fbb69ffb700 1 --2- 172.21.15.142:0/2070288261 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fbb5c0a3c40 0x7fbb5c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:26.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.566+0000 7fbb69ffb700 1 -- 172.21.15.142:0/2070288261 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6723+0+0 (secure 0 0 0) 0x7fbb600d14a0 con 0x7fbb6c103490 2024-04-26T21:31:26.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.566+0000 7fbb6bfff700 1 --2- 172.21.15.142:0/2070288261 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fbb5c0a3c40 0x7fbb5c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:26.568 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.567+0000 7fbb6bfff700 1 --2- 172.21.15.142:0/2070288261 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fbb5c0a3c40 0x7fbb5c0a6100 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbb58005fd0 tx=0x7fbb58005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:26.570 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.570+0000 7fbb69ffb700 1 -- 172.21.15.142:0/2070288261 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbb60031020 con 0x7fbb6c103490 2024-04-26T21:31:26.778 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:26.777+0000 7fbb72ab7700 1 -- 172.21.15.142:0/2070288261 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fbb6c063920 con 0x7fbb5c0a3c40 2024-04-26T21:31:27.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:26 smithi142 ceph-mon[32944]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T21:31:27.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:26 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:27.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:26 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:27.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:26 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:27.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:26 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:27.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:26 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:27.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:26 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:27.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:26 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:27.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:26 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:27.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:26 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:31:27.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:26 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:27.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:26 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:31:27.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:26 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:27.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:26 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:31:27.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:26 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:27.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:26 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:31:27.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:26 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:31:27.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:26 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:27.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:26 smithi050 ceph-mon[32753]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T21:31:27.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:27.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:27.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:27.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:27.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:27.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:27.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:27.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:27.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:31:27.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:27.120 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:31:27.120 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:27.120 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:31:27.120 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:27.120 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:31:27.120 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:31:27.120 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:26 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:27.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:26 smithi046 ceph-mon[29201]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T21:31:27.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:26 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:27.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:26 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:27.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:26 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:27.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:26 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:27.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:26 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:27.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:26 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:27.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:26 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:27.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:26 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:27.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:26 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:31:27.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:26 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:27.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:26 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:31:27.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:26 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:27.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:26 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:31:27.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:26 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:27.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:26 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:31:27.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:26 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:31:27.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:26 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:28.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:27 smithi142 ceph-mon[32944]: Detected new or changed devices on smithi050 2024-04-26T21:31:28.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:27 smithi142 ceph-mon[32944]: Adjusting osd_memory_target on smithi050 to 4273M 2024-04-26T21:31:28.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:27 smithi142 ceph-mon[32944]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:31:28.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:27 smithi142 ceph-mon[32944]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:31:28.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:27 smithi142 ceph-mon[32944]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T21:31:28.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:27 smithi050 ceph-mon[32753]: Detected new or changed devices on smithi050 2024-04-26T21:31:28.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:27 smithi050 ceph-mon[32753]: Adjusting osd_memory_target on smithi050 to 4273M 2024-04-26T21:31:28.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:27 smithi050 ceph-mon[32753]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:31:28.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:27 smithi050 ceph-mon[32753]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:31:28.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:27 smithi050 ceph-mon[32753]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T21:31:28.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:27 smithi046 ceph-mon[29201]: Detected new or changed devices on smithi050 2024-04-26T21:31:28.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:27 smithi046 ceph-mon[29201]: Adjusting osd_memory_target on smithi050 to 4273M 2024-04-26T21:31:28.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:27 smithi046 ceph-mon[29201]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:31:28.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:27 smithi046 ceph-mon[29201]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi142:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:31:28.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:27 smithi046 ceph-mon[29201]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T21:31:29.012 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:28 smithi142 ceph-mon[32944]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:31:29.013 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:28 smithi142 ceph-mon[32944]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T21:31:29.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:28 smithi046 ceph-mon[29201]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:31:29.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:28 smithi046 ceph-mon[29201]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T21:31:29.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:28 smithi050 ceph-mon[32753]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:31:29.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:28 smithi050 ceph-mon[32753]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T21:31:30.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:29 smithi142 ceph-mon[32944]: from='client.? 172.21.15.142:0/1102262636' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b6e3d93-59e1-4853-b4a2-261b728c47a5"}]: dispatch 2024-04-26T21:31:30.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:29 smithi142 ceph-mon[32944]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b6e3d93-59e1-4853-b4a2-261b728c47a5"}]: dispatch 2024-04-26T21:31:30.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:29 smithi142 ceph-mon[32944]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b6e3d93-59e1-4853-b4a2-261b728c47a5"}]': finished 2024-04-26T21:31:30.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:29 smithi142 ceph-mon[32944]: osdmap e77: 12 total, 11 up, 12 in 2024-04-26T21:31:30.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:29 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:31:30.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:29 smithi046 ceph-mon[29201]: from='client.? 172.21.15.142:0/1102262636' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b6e3d93-59e1-4853-b4a2-261b728c47a5"}]: dispatch 2024-04-26T21:31:30.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:29 smithi046 ceph-mon[29201]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b6e3d93-59e1-4853-b4a2-261b728c47a5"}]: dispatch 2024-04-26T21:31:30.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:29 smithi046 ceph-mon[29201]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b6e3d93-59e1-4853-b4a2-261b728c47a5"}]': finished 2024-04-26T21:31:30.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:29 smithi046 ceph-mon[29201]: osdmap e77: 12 total, 11 up, 12 in 2024-04-26T21:31:30.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:29 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:31:30.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:29 smithi050 ceph-mon[32753]: from='client.? 172.21.15.142:0/1102262636' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b6e3d93-59e1-4853-b4a2-261b728c47a5"}]: dispatch 2024-04-26T21:31:30.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:29 smithi050 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b6e3d93-59e1-4853-b4a2-261b728c47a5"}]: dispatch 2024-04-26T21:31:30.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:29 smithi050 ceph-mon[32753]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b6e3d93-59e1-4853-b4a2-261b728c47a5"}]': finished 2024-04-26T21:31:30.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:29 smithi050 ceph-mon[32753]: osdmap e77: 12 total, 11 up, 12 in 2024-04-26T21:31:30.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:29 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:31:31.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:30 smithi142 ceph-mon[32944]: pgmap v247: 1 pgs: 1 remapped+peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:31:31.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:30 smithi142 ceph-mon[32944]: from='client.? 172.21.15.142:0/3411121202' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:31:31.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:30 smithi046 ceph-mon[29201]: pgmap v247: 1 pgs: 1 remapped+peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:31:31.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:30 smithi046 ceph-mon[29201]: from='client.? 172.21.15.142:0/3411121202' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:31:31.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:30 smithi050 ceph-mon[32753]: pgmap v247: 1 pgs: 1 remapped+peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:31:31.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:30 smithi050 ceph-mon[32753]: from='client.? 172.21.15.142:0/3411121202' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:31:33.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:32 smithi142 ceph-mon[32944]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T21:31:33.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:32 smithi046 ceph-mon[29201]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T21:31:33.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:32 smithi050 ceph-mon[32753]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T21:31:34.973 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:34 smithi142 ceph-mon[32944]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-26T21:31:35.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:34 smithi046 ceph-mon[29201]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-26T21:31:35.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:34 smithi050 ceph-mon[32753]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-26T21:31:37.093 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:36 smithi142 ceph-mon[32944]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T21:31:37.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:36 smithi046 ceph-mon[29201]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T21:31:37.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:36 smithi050 ceph-mon[32753]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T21:31:38.038 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:37 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T21:31:38.038 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:37 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:38.038 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:37 smithi142 ceph-mon[32944]: Deploying daemon osd.11 on smithi142 2024-04-26T21:31:38.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:37 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T21:31:38.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:37 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:38.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:37 smithi046 ceph-mon[29201]: Deploying daemon osd.11 on smithi142 2024-04-26T21:31:38.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:37 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T21:31:38.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:37 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:38.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:37 smithi050 ceph-mon[32753]: Deploying daemon osd.11 on smithi142 2024-04-26T21:31:39.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:38 smithi142 ceph-mon[32944]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T21:31:39.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:38 smithi046 ceph-mon[29201]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T21:31:39.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:38 smithi050 ceph-mon[32753]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T21:31:41.179 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:40 smithi142 ceph-mon[32944]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-26T21:31:41.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:40 smithi046 ceph-mon[29201]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-26T21:31:41.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:40 smithi050 ceph-mon[32753]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-26T21:31:43.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:42 smithi142 ceph-mon[32944]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-26T21:31:43.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:42 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:43.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:42 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:43.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:42 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:31:43.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:42 smithi046 ceph-mon[29201]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-26T21:31:43.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:42 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:43.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:42 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:43.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:42 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:31:43.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:42 smithi050 ceph-mon[32753]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-26T21:31:43.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:42 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:43.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:42 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:43.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:42 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:31:45.187 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:44 smithi142 ceph-mon[32944]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:31:45.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:44 smithi046 ceph-mon[29201]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:31:45.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:44 smithi050 ceph-mon[32753]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:31:46.698 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:46.697+0000 7fbb69ffb700 1 -- 172.21.15.142:0/2070288261 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fbb6c063920 con 0x7fbb5c0a3c40 2024-04-26T21:31:46.699 INFO:teuthology.orchestra.run.smithi142.stdout:Created osd(s) 11 on host 'smithi142' 2024-04-26T21:31:46.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:46.701+0000 7fbb72ab7700 1 -- 172.21.15.142:0/2070288261 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fbb5c0a3c40 msgr2=0x7fbb5c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:46.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:46.701+0000 7fbb72ab7700 1 --2- 172.21.15.142:0/2070288261 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fbb5c0a3c40 0x7fbb5c0a6100 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbb58005fd0 tx=0x7fbb58005f60 comp rx=0 tx=0).stop 2024-04-26T21:31:46.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:46.701+0000 7fbb72ab7700 1 -- 172.21.15.142:0/2070288261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb6c103490 msgr2=0x7fbb6c0771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:46.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:46.701+0000 7fbb72ab7700 1 --2- 172.21.15.142:0/2070288261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb6c103490 0x7fbb6c0771c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbb600036f0 tx=0x7fbb60009580 comp rx=0 tx=0).stop 2024-04-26T21:31:46.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:46.701+0000 7fbb72ab7700 1 -- 172.21.15.142:0/2070288261 shutdown_connections 2024-04-26T21:31:46.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:46.701+0000 7fbb72ab7700 1 --2- 172.21.15.142:0/2070288261 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fbb5c0a3c40 0x7fbb5c0a6100 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:46.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:46.701+0000 7fbb72ab7700 1 --2- 172.21.15.142:0/2070288261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb6c105030 0x7fbb6c075d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:46.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:46.701+0000 7fbb72ab7700 1 --2- 172.21.15.142:0/2070288261 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbb6c103490 0x7fbb6c0771c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:46.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:46.701+0000 7fbb72ab7700 1 --2- 172.21.15.142:0/2070288261 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb6c104690 0x7fbb6c075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:46.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:46.701+0000 7fbb72ab7700 1 -- 172.21.15.142:0/2070288261 >> 172.21.15.142:0/2070288261 conn(0x7fbb6c0fea30 msgr2=0x7fbb6c107990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:46.702 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:46.701+0000 7fbb72ab7700 1 -- 172.21.15.142:0/2070288261 shutdown_connections 2024-04-26T21:31:46.703 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:31:46.701+0000 7fbb72ab7700 1 -- 172.21.15.142:0/2070288261 wait complete. 2024-04-26T21:31:47.341 DEBUG:teuthology.orchestra.run.smithi142:osd.11> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.11.service 2024-04-26T21:31:47.344 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-26T21:31:47.344 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-26T21:31:47.602 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:31:47.897 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:47 smithi046 ceph-mon[29201]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:31:47.897 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:47 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:47.898 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:47 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:47.898 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:47 smithi046 ceph-mon[29201]: from='osd.11 [v2:172.21.15.142:6824/1482712612,v1:172.21.15.142:6825/1482712612]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T21:31:47.898 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:47 smithi046 ceph-mon[29201]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T21:31:47.898 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:47 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:47.898 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:47 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:48.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:47 smithi142 ceph-mon[32944]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:31:48.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:47 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:48.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:47 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:48.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:47 smithi142 ceph-mon[32944]: from='osd.11 [v2:172.21.15.142:6824/1482712612,v1:172.21.15.142:6825/1482712612]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T21:31:48.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:47 smithi142 ceph-mon[32944]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T21:31:48.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:47 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:48.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:47 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:48.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:47 smithi050 ceph-mon[32753]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:31:48.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:47 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:48.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:47 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:48.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:47 smithi050 ceph-mon[32753]: from='osd.11 [v2:172.21.15.142:6824/1482712612,v1:172.21.15.142:6825/1482712612]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T21:31:48.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:47 smithi050 ceph-mon[32753]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T21:31:48.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:47 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:48.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:47 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:48.940 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:48 smithi046 ceph-mon[29201]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T21:31:48.940 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:48 smithi046 ceph-mon[29201]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T21:31:48.940 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:48 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:31:48.940 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:48 smithi046 ceph-mon[29201]: from='osd.11 [v2:172.21.15.142:6824/1482712612,v1:172.21.15.142:6825/1482712612]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-04-26T21:31:48.940 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:48 smithi046 ceph-mon[29201]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-04-26T21:31:49.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:48 smithi142 ceph-mon[32944]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T21:31:49.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:48 smithi142 ceph-mon[32944]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T21:31:49.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:48 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:31:49.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:48 smithi142 ceph-mon[32944]: from='osd.11 [v2:172.21.15.142:6824/1482712612,v1:172.21.15.142:6825/1482712612]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-04-26T21:31:49.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:48 smithi142 ceph-mon[32944]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-04-26T21:31:49.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:48 smithi050 ceph-mon[32753]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T21:31:49.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:48 smithi050 ceph-mon[32753]: osdmap e78: 12 total, 11 up, 12 in 2024-04-26T21:31:49.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:48 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:31:49.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:48 smithi050 ceph-mon[32753]: from='osd.11 [v2:172.21.15.142:6824/1482712612,v1:172.21.15.142:6825/1482712612]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-04-26T21:31:49.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:48 smithi050 ceph-mon[32753]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]: dispatch 2024-04-26T21:31:49.401 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:31:49 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:31:49.097+0000 7fc80940b700 -1 osd.11 0 waiting for initial osdmap 2024-04-26T21:31:49.401 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:31:49 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:31:49.116+0000 7fc8011f6700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T21:31:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.510+0000 7f96ce4a1700 1 -- 172.21.15.46:0/4040950097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c8103580 msgr2=0x7f96c81039e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.510+0000 7f96ce4a1700 1 --2- 172.21.15.46:0/4040950097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c8103580 0x7f96c81039e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f96b80099c0 tx=0x7f96b8009cd0 comp rx=0 tx=0).stop 2024-04-26T21:31:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.510+0000 7f96ce4a1700 1 -- 172.21.15.46:0/4040950097 shutdown_connections 2024-04-26T21:31:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.510+0000 7f96ce4a1700 1 --2- 172.21.15.46:0/4040950097 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96c8103f20 0x7f96c810a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.510+0000 7f96ce4a1700 1 --2- 172.21.15.46:0/4040950097 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f96c8102380 0x7f96c81027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.510+0000 7f96ce4a1700 1 --2- 172.21.15.46:0/4040950097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c8103580 0x7f96c81039e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.510+0000 7f96ce4a1700 1 -- 172.21.15.46:0/4040950097 >> 172.21.15.46:0/4040950097 conn(0x7f96c80fd920 msgr2=0x7f96c80ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.510+0000 7f96ce4a1700 1 -- 172.21.15.46:0/4040950097 shutdown_connections 2024-04-26T21:31:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.510+0000 7f96ce4a1700 1 -- 172.21.15.46:0/4040950097 wait complete. 2024-04-26T21:31:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.511+0000 7f96ce4a1700 1 Processor -- start 2024-04-26T21:31:49.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.511+0000 7f96ce4a1700 1 -- start start 2024-04-26T21:31:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.511+0000 7f96ce4a1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96c8102380 0x7f96c8106310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.511+0000 7f96ce4a1700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f96c8103580 0x7f96c8104960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:49.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.511+0000 7f96ce4a1700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c8103f20 0x7f96c8104ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.511+0000 7f96ce4a1700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f96c810de90 con 0x7f96c8103f20 2024-04-26T21:31:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.511+0000 7f96ce4a1700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f96c810dd10 con 0x7f96c8102380 2024-04-26T21:31:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.511+0000 7f96ce4a1700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f96c810e010 con 0x7f96c8103580 2024-04-26T21:31:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.511+0000 7f96cca3e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c8103f20 0x7f96c8104ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.511+0000 7f96cca3e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c8103f20 0x7f96c8104ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37498/0 (socket says 172.21.15.46:37498) 2024-04-26T21:31:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.511+0000 7f96cca3e700 1 -- 172.21.15.46:0/4188606532 learned_addr learned my addr 172.21.15.46:0/4188606532 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:31:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.511+0000 7f96c77fe700 1 --2- 172.21.15.46:0/4188606532 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f96c8103580 0x7f96c8104960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.511+0000 7f96c7fff700 1 --2- 172.21.15.46:0/4188606532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96c8102380 0x7f96c8106310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.512+0000 7f96cca3e700 1 -- 172.21.15.46:0/4188606532 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f96c8103580 msgr2=0x7f96c8104960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.512+0000 7f96cca3e700 1 --2- 172.21.15.46:0/4188606532 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f96c8103580 0x7f96c8104960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.512+0000 7f96cca3e700 1 -- 172.21.15.46:0/4188606532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96c8102380 msgr2=0x7f96c8106310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.512+0000 7f96cca3e700 1 --2- 172.21.15.46:0/4188606532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96c8102380 0x7f96c8106310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:49.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.512+0000 7f96cca3e700 1 -- 172.21.15.46:0/4188606532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96c8105750 con 0x7f96c8103f20 2024-04-26T21:31:49.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.512+0000 7f96cca3e700 1 --2- 172.21.15.46:0/4188606532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c8103f20 0x7f96c8104ea0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f96c0007ee0 tx=0x7f96c000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:49.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.512+0000 7f96c57fa700 1 -- 172.21.15.46:0/4188606532 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c000dbd0 con 0x7f96c8103f20 2024-04-26T21:31:49.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.512+0000 7f96c57fa700 1 -- 172.21.15.46:0/4188606532 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96c000dd70 con 0x7f96c8103f20 2024-04-26T21:31:49.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.512+0000 7f96ce4a1700 1 -- 172.21.15.46:0/4188606532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96c81a37d0 con 0x7f96c8103f20 2024-04-26T21:31:49.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.512+0000 7f96c57fa700 1 -- 172.21.15.46:0/4188606532 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c0005990 con 0x7f96c8103f20 2024-04-26T21:31:49.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.512+0000 7f96ce4a1700 1 -- 172.21.15.46:0/4188606532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96c81a3c90 con 0x7f96c8103f20 2024-04-26T21:31:49.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.513+0000 7f96ce4a1700 1 -- 172.21.15.46:0/4188606532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96c806a110 con 0x7f96c8103f20 2024-04-26T21:31:49.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.514+0000 7f96c57fa700 1 -- 172.21.15.46:0/4188606532 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f96c001a020 con 0x7f96c8103f20 2024-04-26T21:31:49.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.515+0000 7f96c57fa700 1 --2- 172.21.15.46:0/4188606532 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f96b40a3c40 0x7f96b40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:49.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.515+0000 7f96c7fff700 1 --2- 172.21.15.46:0/4188606532 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f96b40a3c40 0x7f96b40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:49.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.515+0000 7f96c57fa700 1 -- 172.21.15.46:0/4188606532 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6847+0+0 (secure 0 0 0) 0x7f96c00cf6a0 con 0x7f96c8103f20 2024-04-26T21:31:49.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.515+0000 7f96c7fff700 1 --2- 172.21.15.46:0/4188606532 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f96b40a3c40 0x7f96b40a6100 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f96b0006fd0 tx=0x7f96b0008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:49.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.520+0000 7f96c57fa700 1 -- 172.21.15.46:0/4188606532 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f96c0010070 con 0x7f96c8103f20 2024-04-26T21:31:49.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.728+0000 7f96ce4a1700 1 -- 172.21.15.46:0/4188606532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f96c81a4530 con 0x7f96c8103f20 2024-04-26T21:31:49.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.728+0000 7f96c57fa700 1 -- 172.21.15.46:0/4188606532 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) ==== 74+0+133 (secure 0 0 0) 0x7f96c009c5a0 con 0x7f96c8103f20 2024-04-26T21:31:49.730 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:31:49.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.730+0000 7f96ce4a1700 1 -- 172.21.15.46:0/4188606532 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f96b40a3c40 msgr2=0x7f96b40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:49.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.730+0000 7f96ce4a1700 1 --2- 172.21.15.46:0/4188606532 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f96b40a3c40 0x7f96b40a6100 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f96b0006fd0 tx=0x7f96b0008040 comp rx=0 tx=0).stop 2024-04-26T21:31:49.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.731+0000 7f96ce4a1700 1 -- 172.21.15.46:0/4188606532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c8103f20 msgr2=0x7f96c8104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:49.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.731+0000 7f96ce4a1700 1 --2- 172.21.15.46:0/4188606532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c8103f20 0x7f96c8104ea0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f96c0007ee0 tx=0x7f96c000d3b0 comp rx=0 tx=0).stop 2024-04-26T21:31:49.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.731+0000 7f96ce4a1700 1 -- 172.21.15.46:0/4188606532 shutdown_connections 2024-04-26T21:31:49.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.731+0000 7f96ce4a1700 1 --2- 172.21.15.46:0/4188606532 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f96b40a3c40 0x7f96b40a6100 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:49.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.731+0000 7f96ce4a1700 1 --2- 172.21.15.46:0/4188606532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c8103f20 0x7f96c8104ea0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:49.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.731+0000 7f96ce4a1700 1 --2- 172.21.15.46:0/4188606532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96c8102380 0x7f96c8106310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:49.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.731+0000 7f96ce4a1700 1 --2- 172.21.15.46:0/4188606532 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f96c8103580 0x7f96c8104960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:49.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.731+0000 7f96ce4a1700 1 -- 172.21.15.46:0/4188606532 >> 172.21.15.46:0/4188606532 conn(0x7f96c80fd920 msgr2=0x7f96c806b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:49.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.731+0000 7f96ce4a1700 1 -- 172.21.15.46:0/4188606532 shutdown_connections 2024-04-26T21:31:49.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:49.731+0000 7f96ce4a1700 1 -- 172.21.15.46:0/4188606532 wait complete. 2024-04-26T21:31:50.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:49 smithi142 ceph-mon[32944]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:31:50.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:49 smithi142 ceph-mon[32944]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-04-26T21:31:50.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:49 smithi142 ceph-mon[32944]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T21:31:50.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:49 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:31:50.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:49 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:31:50.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:49 smithi142 ceph-mon[32944]: from='osd.11 ' entity='osd.11' 2024-04-26T21:31:50.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:49 smithi046 ceph-mon[29201]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:31:50.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:49 smithi046 ceph-mon[29201]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-04-26T21:31:50.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:49 smithi046 ceph-mon[29201]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T21:31:50.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:49 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:31:50.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:49 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:31:50.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:49 smithi046 ceph-mon[29201]: from='osd.11 ' entity='osd.11' 2024-04-26T21:31:50.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:49 smithi050 ceph-mon[32753]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:31:50.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:49 smithi050 ceph-mon[32753]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi142", "root=default"]}]': finished 2024-04-26T21:31:50.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:49 smithi050 ceph-mon[32753]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T21:31:50.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:49 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:31:50.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:49 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:31:50.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:49 smithi050 ceph-mon[32753]: from='osd.11 ' entity='osd.11' 2024-04-26T21:31:50.436 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":79,"num_osds":12,"num_up_osds":11,"osd_up_since":1714167083,"num_in_osds":12,"osd_in_since":1714167089,"num_remapped_pgs":0} 2024-04-26T21:31:50.848 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:50 smithi142 ceph-mon[32944]: purged_snaps scrub starts 2024-04-26T21:31:50.848 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:50 smithi142 ceph-mon[32944]: purged_snaps scrub ok 2024-04-26T21:31:50.848 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:50 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:31:50.848 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:50 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/4188606532' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T21:31:50.848 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:50 smithi142 ceph-mon[32944]: osd.11 [v2:172.21.15.142:6824/1482712612,v1:172.21.15.142:6825/1482712612] boot 2024-04-26T21:31:50.848 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:50 smithi142 ceph-mon[32944]: osdmap e80: 12 total, 12 up, 12 in 2024-04-26T21:31:50.848 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:50 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:31:51.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:50 smithi046 ceph-mon[29201]: purged_snaps scrub starts 2024-04-26T21:31:51.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:50 smithi046 ceph-mon[29201]: purged_snaps scrub ok 2024-04-26T21:31:51.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:50 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:31:51.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:50 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/4188606532' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T21:31:51.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:50 smithi046 ceph-mon[29201]: osd.11 [v2:172.21.15.142:6824/1482712612,v1:172.21.15.142:6825/1482712612] boot 2024-04-26T21:31:51.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:50 smithi046 ceph-mon[29201]: osdmap e80: 12 total, 12 up, 12 in 2024-04-26T21:31:51.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:50 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:31:51.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:50 smithi050 ceph-mon[32753]: purged_snaps scrub starts 2024-04-26T21:31:51.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:50 smithi050 ceph-mon[32753]: purged_snaps scrub ok 2024-04-26T21:31:51.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:50 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:31:51.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:50 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/4188606532' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T21:31:51.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:50 smithi050 ceph-mon[32753]: osd.11 [v2:172.21.15.142:6824/1482712612,v1:172.21.15.142:6825/1482712612] boot 2024-04-26T21:31:51.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:50 smithi050 ceph-mon[32753]: osdmap e80: 12 total, 12 up, 12 in 2024-04-26T21:31:51.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:50 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:31:51.437 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-26T21:31:51.685 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:31:51.717 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:51 smithi046 ceph-mon[29201]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:31:51.717 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:51 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:51.717 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:51 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:51.717 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:51 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:51.717 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:51 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:51.717 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:51 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:51.718 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:51 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:51.718 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:51 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:51.718 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:51 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:51.718 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:51 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:31:51.718 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:51 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:51.718 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:51 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:31:51.718 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:51 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:51.718 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:51 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:31:51.718 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:51 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:52.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:51 smithi142 ceph-mon[32944]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:31:52.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:51 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:52.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:51 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:52.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:51 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:52.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:51 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:52.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:51 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:52.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:51 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:52.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:51 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:52.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:51 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:52.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:51 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:31:52.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:51 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:52.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:51 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:31:52.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:51 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:52.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:51 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:31:52.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:51 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:52.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:51 smithi050 ceph-mon[32753]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:31:52.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:51 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:52.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:51 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:52.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:51 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:52.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:51 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:52.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:51 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:52.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:51 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:31:52.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:51 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:52.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:51 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:52.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:51 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:31:52.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:51 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:52.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:51 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:31:52.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:51 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:52.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:51 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:31:52.120 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:51 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:31:53.040 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:52 smithi046 ceph-mon[29201]: Detected new or changed devices on smithi142 2024-04-26T21:31:53.041 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:52 smithi046 ceph-mon[29201]: Adjusting osd_memory_target on smithi142 to 4272M 2024-04-26T21:31:53.041 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:52 smithi046 ceph-mon[29201]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T21:31:53.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:52 smithi142 ceph-mon[32944]: Detected new or changed devices on smithi142 2024-04-26T21:31:53.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:52 smithi142 ceph-mon[32944]: Adjusting osd_memory_target on smithi142 to 4272M 2024-04-26T21:31:53.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:52 smithi142 ceph-mon[32944]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T21:31:53.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:52 smithi050 ceph-mon[32753]: Detected new or changed devices on smithi142 2024-04-26T21:31:53.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:52 smithi050 ceph-mon[32753]: Adjusting osd_memory_target on smithi142 to 4272M 2024-04-26T21:31:53.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:52 smithi050 ceph-mon[32753]: osdmap e81: 12 total, 12 up, 12 in 2024-04-26T21:31:53.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.630+0000 7f1faad68700 1 -- 172.21.15.46:0/3978525949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fa4104670 msgr2=0x7f1fa4104ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:53.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.630+0000 7f1faad68700 1 --2- 172.21.15.46:0/3978525949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fa4104670 0x7f1fa4104ad0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1f900099c0 tx=0x7f1f90009cd0 comp rx=0 tx=0).stop 2024-04-26T21:31:53.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.630+0000 7f1faad68700 1 -- 172.21.15.46:0/3978525949 shutdown_connections 2024-04-26T21:31:53.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.630+0000 7f1faad68700 1 --2- 172.21.15.46:0/3978525949 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fa4105010 0x7f1fa410b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:53.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.630+0000 7f1faad68700 1 --2- 172.21.15.46:0/3978525949 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1fa4103470 0x7f1fa4103890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:53.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.630+0000 7f1faad68700 1 --2- 172.21.15.46:0/3978525949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fa4104670 0x7f1fa4104ad0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:53.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.630+0000 7f1faad68700 1 -- 172.21.15.46:0/3978525949 >> 172.21.15.46:0/3978525949 conn(0x7f1fa40fea10 msgr2=0x7f1fa4100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:53.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.631+0000 7f1faad68700 1 -- 172.21.15.46:0/3978525949 shutdown_connections 2024-04-26T21:31:53.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.631+0000 7f1faad68700 1 -- 172.21.15.46:0/3978525949 wait complete. 2024-04-26T21:31:53.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.631+0000 7f1faad68700 1 Processor -- start 2024-04-26T21:31:53.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.632+0000 7f1faad68700 1 -- start start 2024-04-26T21:31:53.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.632+0000 7f1faad68700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fa4103470 0x7f1fa4199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:53.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.632+0000 7f1faad68700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1fa4104670 0x7f1fa4199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:53.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.632+0000 7f1faad68700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fa4105010 0x7f1fa41a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:53.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.632+0000 7f1faad68700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f1fa410efb0 con 0x7f1fa4105010 2024-04-26T21:31:53.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.632+0000 7f1faad68700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1fa410ee30 con 0x7f1fa4104670 2024-04-26T21:31:53.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.632+0000 7f1faad68700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f1fa410f130 con 0x7f1fa4103470 2024-04-26T21:31:53.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.632+0000 7f1fa9305700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fa4105010 0x7f1fa41a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:53.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.632+0000 7f1fa3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1fa4104670 0x7f1fa4199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:53.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.632+0000 7f1fa9305700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fa4105010 0x7f1fa41a0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37518/0 (socket says 172.21.15.46:37518) 2024-04-26T21:31:53.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.632+0000 7f1fa3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1fa4104670 0x7f1fa4199b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.46:41050/0 (socket says 172.21.15.46:41050) 2024-04-26T21:31:53.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.632+0000 7f1fa9305700 1 -- 172.21.15.46:0/523528149 learned_addr learned my addr 172.21.15.46:0/523528149 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:31:53.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.632+0000 7f1fa8b04700 1 --2- 172.21.15.46:0/523528149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fa4103470 0x7f1fa4199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:53.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.632+0000 7f1fa9305700 1 -- 172.21.15.46:0/523528149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fa4103470 msgr2=0x7f1fa4199650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:53.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.632+0000 7f1fa9305700 1 --2- 172.21.15.46:0/523528149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fa4103470 0x7f1fa4199650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:53.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.632+0000 7f1fa9305700 1 -- 172.21.15.46:0/523528149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1fa4104670 msgr2=0x7f1fa4199b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:53.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.632+0000 7f1fa9305700 1 --2- 172.21.15.46:0/523528149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1fa4104670 0x7f1fa4199b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:53.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.632+0000 7f1fa9305700 1 -- 172.21.15.46:0/523528149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fa41a12b0 con 0x7f1fa4105010 2024-04-26T21:31:53.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.633+0000 7f1fa9305700 1 --2- 172.21.15.46:0/523528149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fa4105010 0x7f1fa41a0c10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c00b9a0 tx=0x7f1f9c00bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:53.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.633+0000 7f1fa1ffb700 1 -- 172.21.15.46:0/523528149 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c019070 con 0x7f1fa4105010 2024-04-26T21:31:53.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.633+0000 7f1faad68700 1 -- 172.21.15.46:0/523528149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fa41a15a0 con 0x7f1fa4105010 2024-04-26T21:31:53.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.633+0000 7f1fa1ffb700 1 -- 172.21.15.46:0/523528149 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f9c00cde0 con 0x7f1fa4105010 2024-04-26T21:31:53.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.633+0000 7f1fa1ffb700 1 -- 172.21.15.46:0/523528149 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c007370 con 0x7f1fa4105010 2024-04-26T21:31:53.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.633+0000 7f1faad68700 1 -- 172.21.15.46:0/523528149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fa41a1ae0 con 0x7f1fa4105010 2024-04-26T21:31:53.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.634+0000 7f1faad68700 1 -- 172.21.15.46:0/523528149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fa406a110 con 0x7f1fa4105010 2024-04-26T21:31:53.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.635+0000 7f1fa1ffb700 1 -- 172.21.15.46:0/523528149 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1f9c01c070 con 0x7f1fa4105010 2024-04-26T21:31:53.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.636+0000 7f1fa1ffb700 1 --2- 172.21.15.46:0/523528149 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f1f940a3c40 0x7f1f940a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:53.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.636+0000 7f1fa8b04700 1 --2- 172.21.15.46:0/523528149 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f1f940a3c40 0x7f1f940a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:53.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.636+0000 7f1fa1ffb700 1 -- 172.21.15.46:0/523528149 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f1f9c0cf5b0 con 0x7f1fa4105010 2024-04-26T21:31:53.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.636+0000 7f1fa8b04700 1 --2- 172.21.15.46:0/523528149 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f1f940a3c40 0x7f1f940a6100 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1f98005950 tx=0x7f1f980058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:53.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.640+0000 7f1fa1ffb700 1 -- 172.21.15.46:0/523528149 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1f9c09c390 con 0x7f1fa4105010 2024-04-26T21:31:53.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.850+0000 7f1faad68700 1 -- 172.21.15.46:0/523528149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f1fa4064030 con 0x7f1fa4105010 2024-04-26T21:31:53.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.851+0000 7f1fa1ffb700 1 -- 172.21.15.46:0/523528149 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7f1f9c09bc90 con 0x7f1fa4105010 2024-04-26T21:31:53.852 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:31:53.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.853+0000 7f1faad68700 1 -- 172.21.15.46:0/523528149 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f1f940a3c40 msgr2=0x7f1f940a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:53.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.853+0000 7f1faad68700 1 --2- 172.21.15.46:0/523528149 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f1f940a3c40 0x7f1f940a6100 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1f98005950 tx=0x7f1f980058e0 comp rx=0 tx=0).stop 2024-04-26T21:31:53.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.853+0000 7f1faad68700 1 -- 172.21.15.46:0/523528149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fa4105010 msgr2=0x7f1fa41a0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:53.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.853+0000 7f1faad68700 1 --2- 172.21.15.46:0/523528149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fa4105010 0x7f1fa41a0c10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c00b9a0 tx=0x7f1f9c00bcb0 comp rx=0 tx=0).stop 2024-04-26T21:31:53.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.853+0000 7f1faad68700 1 -- 172.21.15.46:0/523528149 shutdown_connections 2024-04-26T21:31:53.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.853+0000 7f1faad68700 1 --2- 172.21.15.46:0/523528149 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f1f940a3c40 0x7f1f940a6100 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:53.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.853+0000 7f1faad68700 1 --2- 172.21.15.46:0/523528149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fa4105010 0x7f1fa41a0c10 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:53.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.853+0000 7f1faad68700 1 --2- 172.21.15.46:0/523528149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1fa4103470 0x7f1fa4199650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:53.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.853+0000 7f1faad68700 1 --2- 172.21.15.46:0/523528149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1fa4104670 0x7f1fa4199b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:53.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.853+0000 7f1faad68700 1 -- 172.21.15.46:0/523528149 >> 172.21.15.46:0/523528149 conn(0x7f1fa40fea10 msgr2=0x7f1fa41079f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:53.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.853+0000 7f1faad68700 1 -- 172.21.15.46:0/523528149 shutdown_connections 2024-04-26T21:31:53.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:53.853+0000 7f1faad68700 1 -- 172.21.15.46:0/523528149 wait complete. 2024-04-26T21:31:53.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:53 smithi050 ceph-mon[32753]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:31:53.946 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:53 smithi142 ceph-mon[32944]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:31:54.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:53 smithi046 ceph-mon[29201]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:31:54.641 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1714167110,"num_in_osds":12,"osd_in_since":1714167089,"num_remapped_pgs":0} 2024-04-26T21:31:54.642 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T21:31:54.895 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:31:55.017 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:54 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/523528149' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T21:31:55.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:54 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/523528149' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T21:31:55.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:54 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/523528149' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T21:31:56.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:55 smithi142 ceph-mon[32944]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:31:56.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:55 smithi046 ceph-mon[29201]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:31:56.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:55 smithi050 ceph-mon[32753]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:31:56.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.854+0000 7f79605af700 1 -- 172.21.15.46:0/3548928954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f795810fe20 msgr2=0x7f7958112210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:56.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.854+0000 7f79605af700 1 --2- 172.21.15.46:0/3548928954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f795810fe20 0x7f7958112210 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f795000b0a0 tx=0x7f795000b3b0 comp rx=0 tx=0).stop 2024-04-26T21:31:56.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.854+0000 7f79605af700 1 -- 172.21.15.46:0/3548928954 shutdown_connections 2024-04-26T21:31:56.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.854+0000 7f79605af700 1 --2- 172.21.15.46:0/3548928954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f795810fe20 0x7f7958112210 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:56.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.854+0000 7f79605af700 1 --2- 172.21.15.46:0/3548928954 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7958075650 0x7f7958075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:56.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.854+0000 7f79605af700 1 --2- 172.21.15.46:0/3548928954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7958075f70 0x7f795810f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:56.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.854+0000 7f79605af700 1 -- 172.21.15.46:0/3548928954 >> 172.21.15.46:0/3548928954 conn(0x7f795807a580 msgr2=0x7f795807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:56.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.854+0000 7f79605af700 1 -- 172.21.15.46:0/3548928954 shutdown_connections 2024-04-26T21:31:56.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.854+0000 7f79605af700 1 -- 172.21.15.46:0/3548928954 wait complete. 2024-04-26T21:31:56.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.854+0000 7f79605af700 1 Processor -- start 2024-04-26T21:31:56.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.855+0000 7f79605af700 1 -- start start 2024-04-26T21:31:56.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.855+0000 7f79605af700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7958075650 0x7f79581a1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:56.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.855+0000 7f79605af700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7958075f70 0x7f79581a1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:56.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.855+0000 7f79605af700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f795810fe20 0x7f795819b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:56.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.855+0000 7f79605af700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f79581157d0 con 0x7f795810fe20 2024-04-26T21:31:56.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.855+0000 7f79605af700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7958115650 con 0x7f7958075f70 2024-04-26T21:31:56.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.855+0000 7f79605af700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7958115950 con 0x7f7958075650 2024-04-26T21:31:56.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.855+0000 7f795eb4c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f795810fe20 0x7f795819b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:56.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.855+0000 7f795eb4c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f795810fe20 0x7f795819b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43520/0 (socket says 172.21.15.46:43520) 2024-04-26T21:31:56.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.855+0000 7f795eb4c700 1 -- 172.21.15.46:0/3580259817 learned_addr learned my addr 172.21.15.46:0/3580259817 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:31:56.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.855+0000 7f795db4a700 1 --2- 172.21.15.46:0/3580259817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7958075f70 0x7f79581a1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:56.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.855+0000 7f795eb4c700 1 -- 172.21.15.46:0/3580259817 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7958075650 msgr2=0x7f79581a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:56.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.855+0000 7f795e34b700 1 --2- 172.21.15.46:0/3580259817 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7958075650 0x7f79581a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:56.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.855+0000 7f795eb4c700 1 --2- 172.21.15.46:0/3580259817 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7958075650 0x7f79581a1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:56.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.855+0000 7f795eb4c700 1 -- 172.21.15.46:0/3580259817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7958075f70 msgr2=0x7f79581a1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:56.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.855+0000 7f795eb4c700 1 --2- 172.21.15.46:0/3580259817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7958075f70 0x7f79581a1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:56.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.855+0000 7f795eb4c700 1 -- 172.21.15.46:0/3580259817 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f795819c130 con 0x7f795810fe20 2024-04-26T21:31:56.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.855+0000 7f795eb4c700 1 --2- 172.21.15.46:0/3580259817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f795810fe20 0x7f795819b9d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7950006070 tx=0x7f7950004f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:56.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.855+0000 7f794f7fe700 1 -- 172.21.15.46:0/3580259817 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7950021070 con 0x7f795810fe20 2024-04-26T21:31:56.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.856+0000 7f79605af700 1 -- 172.21.15.46:0/3580259817 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f795819c3c0 con 0x7f795810fe20 2024-04-26T21:31:56.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.856+0000 7f794f7fe700 1 -- 172.21.15.46:0/3580259817 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7950007d70 con 0x7f795810fe20 2024-04-26T21:31:56.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.856+0000 7f794f7fe700 1 -- 172.21.15.46:0/3580259817 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795001d450 con 0x7f795810fe20 2024-04-26T21:31:56.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.856+0000 7f79605af700 1 -- 172.21.15.46:0/3580259817 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79581a86e0 con 0x7f795810fe20 2024-04-26T21:31:56.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.857+0000 7f79605af700 1 -- 172.21.15.46:0/3580259817 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f795806a110 con 0x7f795810fe20 2024-04-26T21:31:56.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.858+0000 7f794f7fe700 1 -- 172.21.15.46:0/3580259817 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7950004000 con 0x7f795810fe20 2024-04-26T21:31:56.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.858+0000 7f794f7fe700 1 --2- 172.21.15.46:0/3580259817 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f79440a3c40 0x7f79440a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:56.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.858+0000 7f794f7fe700 1 -- 172.21.15.46:0/3580259817 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f795000f020 con 0x7f795810fe20 2024-04-26T21:31:56.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.858+0000 7f795e34b700 1 --2- 172.21.15.46:0/3580259817 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f79440a3c40 0x7f79440a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:56.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.859+0000 7f795e34b700 1 --2- 172.21.15.46:0/3580259817 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f79440a3c40 0x7f79440a6100 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f795819cb10 tx=0x7f7954005d70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:56.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:56.862+0000 7f794f7fe700 1 -- 172.21.15.46:0/3580259817 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f795002a020 con 0x7f795810fe20 2024-04-26T21:31:57.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:57.072+0000 7f79605af700 1 -- 172.21.15.46:0/3580259817 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f79581a88b0 con 0x7f795810fe20 2024-04-26T21:31:57.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:57.073+0000 7f794f7fe700 1 -- 172.21.15.46:0/3580259817 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18434 (secure 0 0 0) 0x7f795009e990 con 0x7f795810fe20 2024-04-26T21:31:57.074 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:31:57.074 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":81,"fsid":"36c7013a-0413-11ef-bc93-c7b262605968","created":"2024-04-26T21:24:30.481156+0000","modified":"2024-04-26T21:31:51.663297+0000","last_up_change":"2024-04-26T21:31:50.100491+0000","last_in_change":"2024-04-26T21:31:29.643428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:27:57.935988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f53c8702-e4da-435f-a9dc-3911d0d36ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6803","nonce":2762665125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6805","nonce":2762665125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6809","nonce":2762665125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6807","nonce":2762665125}]},"public_addr":"172.21.15.46:6803/2762665125","cluster_addr":"172.21.15.46:6805/2762665125","heartbeat_back_addr":"172.21.15.46:6809/2762665125","heartbeat_front_addr":"172.21.15.46:6807/2762665125","state":["exists","up"]},{"osd":1,"uuid":"fc4d6f52-fc8a-4efd-80c9-c092b57e7f42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6801","nonce":1628008970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6803","nonce":1628008970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6807","nonce":1628008970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6805","nonce":1628008970}]},"public_addr":"172.21.15.50:6801/1628008970","cluster_addr":"172.21.15.50:6803/1628008970","heartbeat_back_addr":"172.21.15.50:6807/1628008970","heartbeat_front_addr":"172.21.15.50:6805/1628008970","state":["exists","up"]},{"osd":2,"uuid":"26d1baa1-4ea1-4b7e-9fcf-4f8bbf7329c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6801","nonce":1513279137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6803","nonce":1513279137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6807","nonce":1513279137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6805","nonce":1513279137}]},"public_addr":"172.21.15.142:6801/1513279137","cluster_addr":"172.21.15.142:6803/1513279137","heartbeat_back_addr":"172.21.15.142:6807/1513279137","heartbeat_front_addr":"172.21.15.142:6805/1513279137","state":["exists","up"]},{"osd":3,"uuid":"34d2401f-5fcc-491c-8fcf-c3c492611f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6811","nonce":2435181760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6813","nonce":2435181760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6817","nonce":2435181760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6815","nonce":2435181760}]},"public_addr":"172.21.15.46:6811/2435181760","cluster_addr":"172.21.15.46:6813/2435181760","heartbeat_back_addr":"172.21.15.46:6817/2435181760","heartbeat_front_addr":"172.21.15.46:6815/2435181760","state":["exists","up"]},{"osd":4,"uuid":"986bb709-e4cf-400d-982e-c3f5ac413e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6809","nonce":3177333250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6811","nonce":3177333250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6815","nonce":3177333250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6813","nonce":3177333250}]},"public_addr":"172.21.15.50:6809/3177333250","cluster_addr":"172.21.15.50:6811/3177333250","heartbeat_back_addr":"172.21.15.50:6815/3177333250","heartbeat_front_addr":"172.21.15.50:6813/3177333250","state":["exists","up"]},{"osd":5,"uuid":"b7d65851-ebe1-45e2-833d-8022446f2698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6809","nonce":4113107737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6811","nonce":4113107737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6815","nonce":4113107737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6813","nonce":4113107737}]},"public_addr":"172.21.15.142:6809/4113107737","cluster_addr":"172.21.15.142:6811/4113107737","heartbeat_back_addr":"172.21.15.142:6815/4113107737","heartbeat_front_addr":"172.21.15.142:6813/4113107737","state":["exists","up"]},{"osd":6,"uuid":"2a3e96d7-d5bf-44ff-9134-29f1420c0e4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6819","nonce":2325162882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6821","nonce":2325162882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6825","nonce":2325162882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6823","nonce":2325162882}]},"public_addr":"172.21.15.46:6819/2325162882","cluster_addr":"172.21.15.46:6821/2325162882","heartbeat_back_addr":"172.21.15.46:6825/2325162882","heartbeat_front_addr":"172.21.15.46:6823/2325162882","state":["exists","up"]},{"osd":7,"uuid":"e53ae885-23b6-46fd-b4b1-c62f31a6921f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6817","nonce":820310228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6819","nonce":820310228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6823","nonce":820310228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6821","nonce":820310228}]},"public_addr":"172.21.15.50:6817/820310228","cluster_addr":"172.21.15.50:6819/820310228","heartbeat_back_addr":"172.21.15.50:6823/820310228","heartbeat_front_addr":"172.21.15.50:6821/820310228","state":["exists","up"]},{"osd":8,"uuid":"356e2692-16a9-4045-b58f-c8f9cb5d7241","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6817","nonce":1824918711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6819","nonce":1824918711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6823","nonce":1824918711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6821","nonce":1824918711}]},"public_addr":"172.21.15.142:6817/1824918711","cluster_addr":"172.21.15.142:6819/1824918711","heartbeat_back_addr":"172.21.15.142:6823/1824918711","heartbeat_front_addr":"172.21.15.142:6821/1824918711","state":["exists","up"]},{"osd":9,"uuid":"8e55570e-77e5-4892-8b9f-77dcd03a4eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6827","nonce":3601947736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6829","nonce":3601947736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6833","nonce":3601947736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6831","nonce":3601947736}]},"public_addr":"172.21.15.46:6827/3601947736","cluster_addr":"172.21.15.46:6829/3601947736","heartbeat_back_addr":"172.21.15.46:6833/3601947736","heartbeat_front_addr":"172.21.15.46:6831/3601947736","state":["exists","up"]},{"osd":10,"uuid":"366389fa-fb9a-408b-b5c2-343c054c5a64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6825","nonce":3135765006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6827","nonce":3135765006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6831","nonce":3135765006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6829","nonce":3135765006}]},"public_addr":"172.21.15.50:6825/3135765006","cluster_addr":"172.21.15.50:6827/3135765006","heartbeat_back_addr":"172.21.15.50:6831/3135765006","heartbeat_front_addr":"172.21.15.50:6829/3135765006","state":["exists","up"]},{"osd":11,"uuid":"7b6e3d93-59e1-4853-b4a2-261b728c47a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6825","nonce":1482712612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6827","nonce":1482712612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6831","nonce":1482712612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6829","nonce":1482712612}]},"public_addr":"172.21.15.142:6825/1482712612","cluster_addr":"172.21.15.142:6827/1482712612","heartbeat_back_addr":"172.21.15.142:6831/1482712612","heartbeat_front_addr":"172.21.15.142:6829/1482712612","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:08.082015+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:32.634928+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:55.978864+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:21.796863+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:46.862518+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:11.440432+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:37.764777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:03.374958+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:28.714052+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:55.761167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:22.330951+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:47.839334+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/2465242088":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6801/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6800/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/1381997090":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/3507268217":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2570024356":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/2691061938":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2775954801":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6800/2896098005":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6801/2896098005":"2024-04-27T21:25:02.078730+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:31:57.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:57.075+0000 7f79605af700 1 -- 172.21.15.46:0/3580259817 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f79440a3c40 msgr2=0x7f79440a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:57.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:57.075+0000 7f79605af700 1 --2- 172.21.15.46:0/3580259817 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f79440a3c40 0x7f79440a6100 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f795819cb10 tx=0x7f7954005d70 comp rx=0 tx=0).stop 2024-04-26T21:31:57.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:57.075+0000 7f79605af700 1 -- 172.21.15.46:0/3580259817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f795810fe20 msgr2=0x7f795819b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:57.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:57.075+0000 7f79605af700 1 --2- 172.21.15.46:0/3580259817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f795810fe20 0x7f795819b9d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7950006070 tx=0x7f7950004f10 comp rx=0 tx=0).stop 2024-04-26T21:31:57.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:57.075+0000 7f79605af700 1 -- 172.21.15.46:0/3580259817 shutdown_connections 2024-04-26T21:31:57.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:57.075+0000 7f79605af700 1 --2- 172.21.15.46:0/3580259817 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f79440a3c40 0x7f79440a6100 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:57.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:57.075+0000 7f79605af700 1 --2- 172.21.15.46:0/3580259817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f795810fe20 0x7f795819b9d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:57.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:57.075+0000 7f79605af700 1 --2- 172.21.15.46:0/3580259817 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7958075650 0x7f79581a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:57.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:57.075+0000 7f79605af700 1 --2- 172.21.15.46:0/3580259817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7958075f70 0x7f79581a1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:57.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:57.075+0000 7f79605af700 1 -- 172.21.15.46:0/3580259817 >> 172.21.15.46:0/3580259817 conn(0x7f795807a580 msgr2=0x7f7958100550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:57.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:57.075+0000 7f79605af700 1 -- 172.21.15.46:0/3580259817 shutdown_connections 2024-04-26T21:31:57.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:31:57.075+0000 7f79605af700 1 -- 172.21.15.46:0/3580259817 wait complete. 2024-04-26T21:31:57.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:57 smithi046 ceph-mon[29201]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:31:57.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:57 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3580259817' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:31:57.920 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T21:27:57.935988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T21:31:57.920 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-26T21:31:58.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:57 smithi142 ceph-mon[32944]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:31:58.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:57 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3580259817' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:31:58.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:57 smithi050 ceph-mon[32753]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:31:58.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:57 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3580259817' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:31:58.171 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:31:59.785 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:31:59 smithi046 ceph-mon[29201]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:00.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.033+0000 7fc65e16c700 1 -- 172.21.15.46:0/3163350742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc658102320 msgr2=0x7fc6581027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:00.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.033+0000 7fc65e16c700 1 --2- 172.21.15.46:0/3163350742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc658102320 0x7fc6581027a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc64c009990 tx=0x7fc64c009ca0 comp rx=0 tx=0).stop 2024-04-26T21:32:00.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.033+0000 7fc65e16c700 1 -- 172.21.15.46:0/3163350742 shutdown_connections 2024-04-26T21:32:00.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.033+0000 7fc65e16c700 1 --2- 172.21.15.46:0/3163350742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc658102ce0 0x7fc65810f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:00.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.033+0000 7fc65e16c700 1 --2- 172.21.15.46:0/3163350742 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc658108390 0x7fc658108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:00.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.033+0000 7fc65e16c700 1 --2- 172.21.15.46:0/3163350742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc658102320 0x7fc6581027a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:00.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.033+0000 7fc65e16c700 1 -- 172.21.15.46:0/3163350742 >> 172.21.15.46:0/3163350742 conn(0x7fc6580792d0 msgr2=0x7fc6580796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:00.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.033+0000 7fc65e16c700 1 -- 172.21.15.46:0/3163350742 shutdown_connections 2024-04-26T21:32:00.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.033+0000 7fc65e16c700 1 -- 172.21.15.46:0/3163350742 wait complete. 2024-04-26T21:32:00.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.034+0000 7fc65e16c700 1 Processor -- start 2024-04-26T21:32:00.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.034+0000 7fc65e16c700 1 -- start start 2024-04-26T21:32:00.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.034+0000 7fc65e16c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc658102320 0x7fc658104000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:00.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.034+0000 7fc65e16c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc658102ce0 0x7fc658104540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:00.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.034+0000 7fc65e16c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc658108390 0x7fc658104a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:00.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.034+0000 7fc65e16c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc6581124b0 con 0x7fc658108390 2024-04-26T21:32:00.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.034+0000 7fc65e16c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc658112330 con 0x7fc658102320 2024-04-26T21:32:00.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.034+0000 7fc65e16c700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc658112630 con 0x7fc658102ce0 2024-04-26T21:32:00.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.034+0000 7fc657fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc658108390 0x7fc658104a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:00.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.034+0000 7fc656ffd700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc658102ce0 0x7fc658104540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:00.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.034+0000 7fc6577fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc658102320 0x7fc658104000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:00.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.035+0000 7fc657fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc658108390 0x7fc658104a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43534/0 (socket says 172.21.15.46:43534) 2024-04-26T21:32:00.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.035+0000 7fc657fff700 1 -- 172.21.15.46:0/672482114 learned_addr learned my addr 172.21.15.46:0/672482114 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:00.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.035+0000 7fc6577fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc658102320 0x7fc658104000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.46:33522/0 (socket says 172.21.15.46:33522) 2024-04-26T21:32:00.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.035+0000 7fc657fff700 1 -- 172.21.15.46:0/672482114 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc658102ce0 msgr2=0x7fc658104540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:00.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.035+0000 7fc657fff700 1 --2- 172.21.15.46:0/672482114 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc658102ce0 0x7fc658104540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:00.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.035+0000 7fc657fff700 1 -- 172.21.15.46:0/672482114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc658102320 msgr2=0x7fc658104000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:00.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.035+0000 7fc657fff700 1 --2- 172.21.15.46:0/672482114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc658102320 0x7fc658104000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:00.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.035+0000 7fc657fff700 1 -- 172.21.15.46:0/672482114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6581a7f70 con 0x7fc658108390 2024-04-26T21:32:00.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.035+0000 7fc656ffd700 1 --2- 172.21.15.46:0/672482114 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc658102ce0 0x7fc658104540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:32:00.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.035+0000 7fc657fff700 1 --2- 172.21.15.46:0/672482114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc658108390 0x7fc658104a80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc650007d70 tx=0x7fc65000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:00.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.035+0000 7fc654ff9700 1 -- 172.21.15.46:0/672482114 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc650016020 con 0x7fc658108390 2024-04-26T21:32:00.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.035+0000 7fc65e16c700 1 -- 172.21.15.46:0/672482114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6581a8260 con 0x7fc658108390 2024-04-26T21:32:00.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.035+0000 7fc654ff9700 1 -- 172.21.15.46:0/672482114 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc65000ac90 con 0x7fc658108390 2024-04-26T21:32:00.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.035+0000 7fc654ff9700 1 -- 172.21.15.46:0/672482114 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6500115e0 con 0x7fc658108390 2024-04-26T21:32:00.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.035+0000 7fc65e16c700 1 -- 172.21.15.46:0/672482114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6581a8720 con 0x7fc658108390 2024-04-26T21:32:00.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.037+0000 7fc65e16c700 1 -- 172.21.15.46:0/672482114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc658103a40 con 0x7fc658108390 2024-04-26T21:32:00.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.038+0000 7fc654ff9700 1 -- 172.21.15.46:0/672482114 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc650019020 con 0x7fc658108390 2024-04-26T21:32:00.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.039+0000 7fc654ff9700 1 --2- 172.21.15.46:0/672482114 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fc6400a3c40 0x7fc6400a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:00.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.039+0000 7fc6577fe700 1 --2- 172.21.15.46:0/672482114 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fc6400a3c40 0x7fc6400a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:00.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.039+0000 7fc654ff9700 1 -- 172.21.15.46:0/672482114 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc6500cfef0 con 0x7fc658108390 2024-04-26T21:32:00.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.039+0000 7fc6577fe700 1 --2- 172.21.15.46:0/672482114 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fc6400a3c40 0x7fc6400a6100 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc648005950 tx=0x7fc64800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:00.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:31:59 smithi142 ceph-mon[32944]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:00.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.043+0000 7fc654ff9700 1 -- 172.21.15.46:0/672482114 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc650020020 con 0x7fc658108390 2024-04-26T21:32:00.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:31:59 smithi050 ceph-mon[32753]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:00.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.249+0000 7fc65e16c700 1 -- 172.21.15.46:0/672482114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc6581a8a10 con 0x7fc658108390 2024-04-26T21:32:00.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.249+0000 7fc654ff9700 1 -- 172.21.15.46:0/672482114 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fc65009cd70 con 0x7fc658108390 2024-04-26T21:32:00.250 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-26T21:32:00.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.251+0000 7fc65e16c700 1 -- 172.21.15.46:0/672482114 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fc6400a3c40 msgr2=0x7fc6400a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:00.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.251+0000 7fc65e16c700 1 --2- 172.21.15.46:0/672482114 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fc6400a3c40 0x7fc6400a6100 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc648005950 tx=0x7fc64800b410 comp rx=0 tx=0).stop 2024-04-26T21:32:00.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.251+0000 7fc65e16c700 1 -- 172.21.15.46:0/672482114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc658108390 msgr2=0x7fc658104a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:00.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.251+0000 7fc65e16c700 1 --2- 172.21.15.46:0/672482114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc658108390 0x7fc658104a80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc650007d70 tx=0x7fc65000a340 comp rx=0 tx=0).stop 2024-04-26T21:32:00.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.251+0000 7fc65e16c700 1 -- 172.21.15.46:0/672482114 shutdown_connections 2024-04-26T21:32:00.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.251+0000 7fc65e16c700 1 --2- 172.21.15.46:0/672482114 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fc6400a3c40 0x7fc6400a6100 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:00.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.251+0000 7fc65e16c700 1 --2- 172.21.15.46:0/672482114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc658108390 0x7fc658104a80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:00.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.251+0000 7fc65e16c700 1 --2- 172.21.15.46:0/672482114 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc658102320 0x7fc658104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:00.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.251+0000 7fc65e16c700 1 --2- 172.21.15.46:0/672482114 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc658102ce0 0x7fc658104540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:00.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.251+0000 7fc65e16c700 1 -- 172.21.15.46:0/672482114 >> 172.21.15.46:0/672482114 conn(0x7fc6580792d0 msgr2=0x7fc658101590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:00.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.251+0000 7fc65e16c700 1 -- 172.21.15.46:0/672482114 shutdown_connections 2024-04-26T21:32:00.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:00.251+0000 7fc65e16c700 1 -- 172.21.15.46:0/672482114 wait complete. 2024-04-26T21:32:00.908 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:00 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/672482114' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:32:00.937 INFO:tasks.cephadm:Adding mds.a on smithi046 2024-04-26T21:32:00.937 INFO:tasks.cephadm:Adding mds.d on smithi046 2024-04-26T21:32:00.937 INFO:tasks.cephadm:Adding mds.g on smithi046 2024-04-26T21:32:00.937 INFO:tasks.cephadm:Adding mds.j on smithi046 2024-04-26T21:32:00.937 INFO:tasks.cephadm:Adding mds.b on smithi050 2024-04-26T21:32:00.937 INFO:tasks.cephadm:Adding mds.e on smithi050 2024-04-26T21:32:00.937 INFO:tasks.cephadm:Adding mds.h on smithi050 2024-04-26T21:32:00.937 INFO:tasks.cephadm:Adding mds.k on smithi050 2024-04-26T21:32:00.937 INFO:tasks.cephadm:Adding mds.c on smithi142 2024-04-26T21:32:00.937 INFO:tasks.cephadm:Adding mds.f on smithi142 2024-04-26T21:32:00.937 INFO:tasks.cephadm:Adding mds.i on smithi142 2024-04-26T21:32:00.937 INFO:tasks.cephadm:Adding mds.l on smithi142 2024-04-26T21:32:00.938 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi050=b;smithi050=e;smithi050=h;smithi050=k;smithi142=c;smithi142=f;smithi142=i;smithi142=l' 2024-04-26T21:32:00.965 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:00 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/672482114' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:32:01.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:00 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/672482114' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:32:01.186 INFO:teuthology.orchestra.run.smithi142.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:32:02.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:01 smithi142 ceph-mon[32944]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:02.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:01 smithi046 ceph-mon[29201]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:02.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:01 smithi050 ceph-mon[32753]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:02.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.888+0000 7f4b9cfa4700 1 -- 172.21.15.142:0/1695463884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b98102380 msgr2=0x7f4b981027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:02.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.888+0000 7f4b9cfa4700 1 --2- 172.21.15.142:0/1695463884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b98102380 0x7f4b981027a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4b88009990 tx=0x7f4b88009ca0 comp rx=0 tx=0).stop 2024-04-26T21:32:02.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.888+0000 7f4b9cfa4700 1 -- 172.21.15.142:0/1695463884 shutdown_connections 2024-04-26T21:32:02.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.888+0000 7f4b9cfa4700 1 --2- 172.21.15.142:0/1695463884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b98103f20 0x7f4b9810a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:02.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.888+0000 7f4b9cfa4700 1 --2- 172.21.15.142:0/1695463884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b98102380 0x7f4b981027a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:02.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.888+0000 7f4b9cfa4700 1 --2- 172.21.15.142:0/1695463884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b98103580 0x7f4b981039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:02.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.888+0000 7f4b9cfa4700 1 -- 172.21.15.142:0/1695463884 >> 172.21.15.142:0/1695463884 conn(0x7f4b980fd940 msgr2=0x7f4b980ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:02.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.888+0000 7f4b9cfa4700 1 -- 172.21.15.142:0/1695463884 shutdown_connections 2024-04-26T21:32:02.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.888+0000 7f4b9cfa4700 1 -- 172.21.15.142:0/1695463884 wait complete. 2024-04-26T21:32:02.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.888+0000 7f4b9cfa4700 1 Processor -- start 2024-04-26T21:32:02.891 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.889+0000 7f4b9cfa4700 1 -- start start 2024-04-26T21:32:02.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.889+0000 7f4b9cfa4700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b98102380 0x7f4b9806c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:02.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.889+0000 7f4b9cfa4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b98103580 0x7f4b9819e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:02.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.889+0000 7f4b9cfa4700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b98103f20 0x7f4b981a09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:02.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.889+0000 7f4b9cfa4700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9810de20 con 0x7f4b98103580 2024-04-26T21:32:02.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.889+0000 7f4b9cfa4700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9810dca0 con 0x7f4b98103f20 2024-04-26T21:32:02.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.889+0000 7f4b9cfa4700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9810dfa0 con 0x7f4b98102380 2024-04-26T21:32:02.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.889+0000 7f4b9659c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b98102380 0x7f4b9806c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:02.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.889+0000 7f4b9659c700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b98102380 0x7f4b9806c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.142:3300/0 says I am v2:172.21.15.142:32770/0 (socket says 172.21.15.142:32770) 2024-04-26T21:32:02.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.889+0000 7f4b9659c700 1 -- 172.21.15.142:0/2159901705 learned_addr learned my addr 172.21.15.142:0/2159901705 (peer_addr_for_me v2:172.21.15.142:0/0) 2024-04-26T21:32:02.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.889+0000 7f4b95d9b700 1 --2- 172.21.15.142:0/2159901705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b98103580 0x7f4b9819e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:02.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.889+0000 7f4b96d9d700 1 --2- 172.21.15.142:0/2159901705 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b98103f20 0x7f4b981a09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:02.892 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.889+0000 7f4b9659c700 1 -- 172.21.15.142:0/2159901705 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b98103f20 msgr2=0x7f4b981a09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:02.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.889+0000 7f4b9659c700 1 --2- 172.21.15.142:0/2159901705 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b98103f20 0x7f4b981a09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:02.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.889+0000 7f4b9659c700 1 -- 172.21.15.142:0/2159901705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b98103580 msgr2=0x7f4b9819e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:02.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.889+0000 7f4b9659c700 1 --2- 172.21.15.142:0/2159901705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b98103580 0x7f4b9819e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:02.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.889+0000 7f4b9659c700 1 -- 172.21.15.142:0/2159901705 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b981a12c0 con 0x7f4b98102380 2024-04-26T21:32:02.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.889+0000 7f4b9659c700 1 --2- 172.21.15.142:0/2159901705 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b98102380 0x7f4b9806c690 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4b88000c00 tx=0x7f4b88009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:02.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.889+0000 7f4b877fe700 1 -- 172.21.15.142:0/2159901705 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b88003c10 con 0x7f4b98102380 2024-04-26T21:32:02.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.889+0000 7f4b877fe700 1 -- 172.21.15.142:0/2159901705 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b88004830 con 0x7f4b98102380 2024-04-26T21:32:02.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.889+0000 7f4b9cfa4700 1 -- 172.21.15.142:0/2159901705 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b981a1550 con 0x7f4b98102380 2024-04-26T21:32:02.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.889+0000 7f4b877fe700 1 -- 172.21.15.142:0/2159901705 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b88005310 con 0x7f4b98102380 2024-04-26T21:32:02.893 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.890+0000 7f4b9cfa4700 1 -- 172.21.15.142:0/2159901705 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b981a1940 con 0x7f4b98102380 2024-04-26T21:32:02.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.890+0000 7f4b9cfa4700 1 -- 172.21.15.142:0/2159901705 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b9806a110 con 0x7f4b98102380 2024-04-26T21:32:02.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.892+0000 7f4b877fe700 1 -- 172.21.15.142:0/2159901705 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4b880054b0 con 0x7f4b98102380 2024-04-26T21:32:02.894 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.892+0000 7f4b877fe700 1 --2- 172.21.15.142:0/2159901705 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4b800a3c40 0x7f4b800a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:02.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.893+0000 7f4b877fe700 1 -- 172.21.15.142:0/2159901705 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f4b880d1430 con 0x7f4b98102380 2024-04-26T21:32:02.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.893+0000 7f4b95d9b700 1 --2- 172.21.15.142:0/2159901705 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4b800a3c40 0x7f4b800a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:02.895 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.893+0000 7f4b95d9b700 1 --2- 172.21.15.142:0/2159901705 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4b800a3c40 0x7f4b800a6100 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4b9806cc10 tx=0x7f4b8c00a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:02.897 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:02.896+0000 7f4b877fe700 1 -- 172.21.15.142:0/2159901705 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4b88013070 con 0x7f4b98102380 2024-04-26T21:32:03.107 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:03.107+0000 7f4b9cfa4700 1 -- 172.21.15.142:0/2159901705 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi050=b;smithi050=e;smithi050=h;smithi050=k;smithi142=c;smithi142=f;smithi142=i;smithi142=l", "target": ["mon-mgr", ""]}) -- 0x7f4b98063b20 con 0x7f4b800a3c40 2024-04-26T21:32:03.116 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:03.115+0000 7f4b877fe700 1 -- 172.21.15.142:0/2159901705 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f4b98063b20 con 0x7f4b800a3c40 2024-04-26T21:32:03.116 INFO:teuthology.orchestra.run.smithi142.stdout:Scheduled mds.all update... 2024-04-26T21:32:03.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:03.117+0000 7f4b9cfa4700 1 -- 172.21.15.142:0/2159901705 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4b800a3c40 msgr2=0x7f4b800a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:03.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:03.117+0000 7f4b9cfa4700 1 --2- 172.21.15.142:0/2159901705 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4b800a3c40 0x7f4b800a6100 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4b9806cc10 tx=0x7f4b8c00a300 comp rx=0 tx=0).stop 2024-04-26T21:32:03.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:03.117+0000 7f4b9cfa4700 1 -- 172.21.15.142:0/2159901705 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b98102380 msgr2=0x7f4b9806c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:03.118 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:03.118+0000 7f4b9cfa4700 1 --2- 172.21.15.142:0/2159901705 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b98102380 0x7f4b9806c690 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4b88000c00 tx=0x7f4b88009580 comp rx=0 tx=0).stop 2024-04-26T21:32:03.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:03.118+0000 7f4b9cfa4700 1 -- 172.21.15.142:0/2159901705 shutdown_connections 2024-04-26T21:32:03.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:03.118+0000 7f4b9cfa4700 1 --2- 172.21.15.142:0/2159901705 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b98103f20 0x7f4b981a09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:03.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:03.118+0000 7f4b9cfa4700 1 --2- 172.21.15.142:0/2159901705 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b98102380 0x7f4b9806c690 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:03.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:03.118+0000 7f4b9cfa4700 1 --2- 172.21.15.142:0/2159901705 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4b800a3c40 0x7f4b800a6100 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:03.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:03.118+0000 7f4b9cfa4700 1 --2- 172.21.15.142:0/2159901705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b98103580 0x7f4b9819e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:03.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:03.118+0000 7f4b9cfa4700 1 -- 172.21.15.142:0/2159901705 >> 172.21.15.142:0/2159901705 conn(0x7f4b980fd940 msgr2=0x7f4b981069f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:03.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:03.118+0000 7f4b9cfa4700 1 -- 172.21.15.142:0/2159901705 shutdown_connections 2024-04-26T21:32:03.119 INFO:teuthology.orchestra.run.smithi142.stderr:2024-04-26T21:32:03.118+0000 7f4b9cfa4700 1 -- 172.21.15.142:0/2159901705 wait complete. 2024-04-26T21:32:03.841 DEBUG:teuthology.orchestra.run.smithi046:mds.a> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.a.service 2024-04-26T21:32:03.844 DEBUG:teuthology.orchestra.run.smithi046:mds.d> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.d.service 2024-04-26T21:32:03.888 DEBUG:teuthology.orchestra.run.smithi046:mds.g> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.g.service 2024-04-26T21:32:03.932 DEBUG:teuthology.orchestra.run.smithi046:mds.j> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.j.service 2024-04-26T21:32:03.953 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:03 smithi046 ceph-mon[29201]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:03.953 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:03 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:03.953 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:03 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:32:03.954 DEBUG:teuthology.orchestra.run.smithi050:mds.b> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.b.service 2024-04-26T21:32:03.957 DEBUG:teuthology.orchestra.run.smithi050:mds.e> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.e.service 2024-04-26T21:32:03.978 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:03 smithi050 ceph-mon[32753]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:03.978 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:03 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:03.978 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:03 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:32:03.979 DEBUG:teuthology.orchestra.run.smithi050:mds.h> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.h.service 2024-04-26T21:32:04.023 DEBUG:teuthology.orchestra.run.smithi050:mds.k> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.k.service 2024-04-26T21:32:04.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:03 smithi142 ceph-mon[32944]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:04.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:03 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:04.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:03 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:32:04.065 DEBUG:teuthology.orchestra.run.smithi142:mds.c> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.c.service 2024-04-26T21:32:04.069 DEBUG:teuthology.orchestra.run.smithi142:mds.f> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.f.service 2024-04-26T21:32:04.112 DEBUG:teuthology.orchestra.run.smithi142:mds.i> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.i.service 2024-04-26T21:32:04.155 DEBUG:teuthology.orchestra.run.smithi142:mds.l> sudo journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.l.service 2024-04-26T21:32:04.198 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-26T21:32:04.198 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:32:04.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.316+0000 7f63ab0ea640 1 -- 172.21.15.46:0/3423954549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63a4101890 msgr2=0x7f63a4101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:04.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.316+0000 7f63ab0ea640 1 --2- 172.21.15.46:0/3423954549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63a4101890 0x7f63a4101cb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6398009900 tx=0x7f639802a410 comp rx=0 tx=0).stop 2024-04-26T21:32:04.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.317+0000 7f63ab0ea640 1 -- 172.21.15.46:0/3423954549 shutdown_connections 2024-04-26T21:32:04.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.317+0000 7f63ab0ea640 1 --2- 172.21.15.46:0/3423954549 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63a4103440 0x7f63a4109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.317+0000 7f63ab0ea640 1 --2- 172.21.15.46:0/3423954549 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63a4102a80 0x7f63a4102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.317+0000 7f63ab0ea640 1 --2- 172.21.15.46:0/3423954549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63a4101890 0x7f63a4101cb0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.317+0000 7f63ab0ea640 1 -- 172.21.15.46:0/3423954549 >> 172.21.15.46:0/3423954549 conn(0x7f63a40fce20 msgr2=0x7f63a40ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.317+0000 7f63ab0ea640 1 -- 172.21.15.46:0/3423954549 shutdown_connections 2024-04-26T21:32:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.317+0000 7f63ab0ea640 1 -- 172.21.15.46:0/3423954549 wait complete. 2024-04-26T21:32:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.317+0000 7f63ab0ea640 1 Processor -- start 2024-04-26T21:32:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.317+0000 7f63ab0ea640 1 -- start start 2024-04-26T21:32:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.317+0000 7f63ab0ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63a4101890 0x7f63a4198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.317+0000 7f63ab0ea640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63a4102a80 0x7f63a4199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.317+0000 7f63ab0ea640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63a4103440 0x7f63a41a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.317+0000 7f63ab0ea640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f63a410e4a0 con 0x7f63a4101890 2024-04-26T21:32:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.317+0000 7f63ab0ea640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f63a410e320 con 0x7f63a4103440 2024-04-26T21:32:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.317+0000 7f63ab0ea640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f63a410e620 con 0x7f63a4102a80 2024-04-26T21:32:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.317+0000 7f63a8e86640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63a4101890 0x7f63a4198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.317+0000 7f63a8e86640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63a4101890 0x7f63a4198be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43556/0 (socket says 172.21.15.46:43556) 2024-04-26T21:32:04.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.317+0000 7f63a8e86640 1 -- 172.21.15.46:0/2933868824 learned_addr learned my addr 172.21.15.46:0/2933868824 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.318+0000 7f63a3fff640 1 --2- 172.21.15.46:0/2933868824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63a4102a80 0x7f63a4199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.318+0000 7f63a9687640 1 --2- 172.21.15.46:0/2933868824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63a4103440 0x7f63a41a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.318+0000 7f63a8e86640 1 -- 172.21.15.46:0/2933868824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63a4102a80 msgr2=0x7f63a4199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.318+0000 7f63a8e86640 1 --2- 172.21.15.46:0/2933868824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63a4102a80 0x7f63a4199120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.318+0000 7f63a8e86640 1 -- 172.21.15.46:0/2933868824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63a4103440 msgr2=0x7f63a41a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.318+0000 7f63a8e86640 1 --2- 172.21.15.46:0/2933868824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63a4103440 0x7f63a41a0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.318+0000 7f63a8e86640 1 -- 172.21.15.46:0/2933868824 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63a41a0800 con 0x7f63a4101890 2024-04-26T21:32:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.318+0000 7f63a8e86640 1 --2- 172.21.15.46:0/2933868824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63a4101890 0x7f63a4198be0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6398006010 tx=0x7f63980042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.318+0000 7f63a1ffb640 1 -- 172.21.15.46:0/2933868824 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639803f030 con 0x7f63a4101890 2024-04-26T21:32:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.318+0000 7f63a1ffb640 1 -- 172.21.15.46:0/2933868824 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63980318f0 con 0x7f63a4101890 2024-04-26T21:32:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.318+0000 7f63ab0ea640 1 -- 172.21.15.46:0/2933868824 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63a41a0a90 con 0x7f63a4101890 2024-04-26T21:32:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.318+0000 7f63a1ffb640 1 -- 172.21.15.46:0/2933868824 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6398035070 con 0x7f63a4101890 2024-04-26T21:32:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.318+0000 7f63ab0ea640 1 -- 172.21.15.46:0/2933868824 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63a41a0fd0 con 0x7f63a4101890 2024-04-26T21:32:04.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.319+0000 7f63ab0ea640 1 -- 172.21.15.46:0/2933868824 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63a4108d20 con 0x7f63a4101890 2024-04-26T21:32:04.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.320+0000 7f63a1ffb640 1 -- 172.21.15.46:0/2933868824 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6398042030 con 0x7f63a4101890 2024-04-26T21:32:04.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.321+0000 7f63a1ffb640 1 --2- 172.21.15.46:0/2933868824 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f63940a3c30 0x7f63940a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:04.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.321+0000 7f63a3fff640 1 --2- 172.21.15.46:0/2933868824 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f63940a3c30 0x7f63940a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:04.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.321+0000 7f63a1ffb640 1 -- 172.21.15.46:0/2933868824 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f63980f0520 con 0x7f63a4101890 2024-04-26T21:32:04.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.321+0000 7f63a3fff640 1 --2- 172.21.15.46:0/2933868824 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f63940a3c30 0x7f63940a60f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f63a419a0f0 tx=0x7f6390005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:04.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.326+0000 7f63a1ffb640 1 -- 172.21.15.46:0/2933868824 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f63980bd300 con 0x7f63a4101890 2024-04-26T21:32:04.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.511+0000 7f63ab0ea640 1 -- 172.21.15.46:0/2933868824 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f63a4063530 con 0x7f63a4101890 2024-04-26T21:32:04.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.512+0000 7f63a1ffb640 1 -- 172.21.15.46:0/2933868824 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18434 (secure 0 0 0) 0x7f63980bcc00 con 0x7f63a4101890 2024-04-26T21:32:04.514 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:32:04.514 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":81,"fsid":"36c7013a-0413-11ef-bc93-c7b262605968","created":"2024-04-26T21:24:30.481156+0000","modified":"2024-04-26T21:31:51.663297+0000","last_up_change":"2024-04-26T21:31:50.100491+0000","last_in_change":"2024-04-26T21:31:29.643428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:27:57.935988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f53c8702-e4da-435f-a9dc-3911d0d36ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6803","nonce":2762665125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6805","nonce":2762665125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6809","nonce":2762665125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6807","nonce":2762665125}]},"public_addr":"172.21.15.46:6803/2762665125","cluster_addr":"172.21.15.46:6805/2762665125","heartbeat_back_addr":"172.21.15.46:6809/2762665125","heartbeat_front_addr":"172.21.15.46:6807/2762665125","state":["exists","up"]},{"osd":1,"uuid":"fc4d6f52-fc8a-4efd-80c9-c092b57e7f42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6801","nonce":1628008970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6803","nonce":1628008970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6807","nonce":1628008970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6805","nonce":1628008970}]},"public_addr":"172.21.15.50:6801/1628008970","cluster_addr":"172.21.15.50:6803/1628008970","heartbeat_back_addr":"172.21.15.50:6807/1628008970","heartbeat_front_addr":"172.21.15.50:6805/1628008970","state":["exists","up"]},{"osd":2,"uuid":"26d1baa1-4ea1-4b7e-9fcf-4f8bbf7329c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6801","nonce":1513279137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6803","nonce":1513279137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6807","nonce":1513279137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6805","nonce":1513279137}]},"public_addr":"172.21.15.142:6801/1513279137","cluster_addr":"172.21.15.142:6803/1513279137","heartbeat_back_addr":"172.21.15.142:6807/1513279137","heartbeat_front_addr":"172.21.15.142:6805/1513279137","state":["exists","up"]},{"osd":3,"uuid":"34d2401f-5fcc-491c-8fcf-c3c492611f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6811","nonce":2435181760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6813","nonce":2435181760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6817","nonce":2435181760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6815","nonce":2435181760}]},"public_addr":"172.21.15.46:6811/2435181760","cluster_addr":"172.21.15.46:6813/2435181760","heartbeat_back_addr":"172.21.15.46:6817/2435181760","heartbeat_front_addr":"172.21.15.46:6815/2435181760","state":["exists","up"]},{"osd":4,"uuid":"986bb709-e4cf-400d-982e-c3f5ac413e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6809","nonce":3177333250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6811","nonce":3177333250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6815","nonce":3177333250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6813","nonce":3177333250}]},"public_addr":"172.21.15.50:6809/3177333250","cluster_addr":"172.21.15.50:6811/3177333250","heartbeat_back_addr":"172.21.15.50:6815/3177333250","heartbeat_front_addr":"172.21.15.50:6813/3177333250","state":["exists","up"]},{"osd":5,"uuid":"b7d65851-ebe1-45e2-833d-8022446f2698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6809","nonce":4113107737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6811","nonce":4113107737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6815","nonce":4113107737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6813","nonce":4113107737}]},"public_addr":"172.21.15.142:6809/4113107737","cluster_addr":"172.21.15.142:6811/4113107737","heartbeat_back_addr":"172.21.15.142:6815/4113107737","heartbeat_front_addr":"172.21.15.142:6813/4113107737","state":["exists","up"]},{"osd":6,"uuid":"2a3e96d7-d5bf-44ff-9134-29f1420c0e4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6819","nonce":2325162882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6821","nonce":2325162882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6825","nonce":2325162882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6823","nonce":2325162882}]},"public_addr":"172.21.15.46:6819/2325162882","cluster_addr":"172.21.15.46:6821/2325162882","heartbeat_back_addr":"172.21.15.46:6825/2325162882","heartbeat_front_addr":"172.21.15.46:6823/2325162882","state":["exists","up"]},{"osd":7,"uuid":"e53ae885-23b6-46fd-b4b1-c62f31a6921f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6817","nonce":820310228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6819","nonce":820310228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6823","nonce":820310228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6821","nonce":820310228}]},"public_addr":"172.21.15.50:6817/820310228","cluster_addr":"172.21.15.50:6819/820310228","heartbeat_back_addr":"172.21.15.50:6823/820310228","heartbeat_front_addr":"172.21.15.50:6821/820310228","state":["exists","up"]},{"osd":8,"uuid":"356e2692-16a9-4045-b58f-c8f9cb5d7241","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6817","nonce":1824918711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6819","nonce":1824918711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6823","nonce":1824918711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6821","nonce":1824918711}]},"public_addr":"172.21.15.142:6817/1824918711","cluster_addr":"172.21.15.142:6819/1824918711","heartbeat_back_addr":"172.21.15.142:6823/1824918711","heartbeat_front_addr":"172.21.15.142:6821/1824918711","state":["exists","up"]},{"osd":9,"uuid":"8e55570e-77e5-4892-8b9f-77dcd03a4eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6827","nonce":3601947736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6829","nonce":3601947736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6833","nonce":3601947736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6831","nonce":3601947736}]},"public_addr":"172.21.15.46:6827/3601947736","cluster_addr":"172.21.15.46:6829/3601947736","heartbeat_back_addr":"172.21.15.46:6833/3601947736","heartbeat_front_addr":"172.21.15.46:6831/3601947736","state":["exists","up"]},{"osd":10,"uuid":"366389fa-fb9a-408b-b5c2-343c054c5a64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6825","nonce":3135765006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6827","nonce":3135765006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6831","nonce":3135765006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6829","nonce":3135765006}]},"public_addr":"172.21.15.50:6825/3135765006","cluster_addr":"172.21.15.50:6827/3135765006","heartbeat_back_addr":"172.21.15.50:6831/3135765006","heartbeat_front_addr":"172.21.15.50:6829/3135765006","state":["exists","up"]},{"osd":11,"uuid":"7b6e3d93-59e1-4853-b4a2-261b728c47a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6825","nonce":1482712612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6827","nonce":1482712612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6831","nonce":1482712612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6829","nonce":1482712612}]},"public_addr":"172.21.15.142:6825/1482712612","cluster_addr":"172.21.15.142:6827/1482712612","heartbeat_back_addr":"172.21.15.142:6831/1482712612","heartbeat_front_addr":"172.21.15.142:6829/1482712612","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:08.082015+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:32.634928+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:55.978864+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:21.796863+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:46.862518+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:11.440432+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:37.764777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:03.374958+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:28.714052+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:55.761167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:22.330951+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:47.839334+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/2465242088":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6801/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6800/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/1381997090":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/3507268217":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2570024356":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/2691061938":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2775954801":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6800/2896098005":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6801/2896098005":"2024-04-27T21:25:02.078730+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:32:04.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.514+0000 7f63ab0ea640 1 -- 172.21.15.46:0/2933868824 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f63940a3c30 msgr2=0x7f63940a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:04.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.514+0000 7f63ab0ea640 1 --2- 172.21.15.46:0/2933868824 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f63940a3c30 0x7f63940a60f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f63a419a0f0 tx=0x7f6390005f90 comp rx=0 tx=0).stop 2024-04-26T21:32:04.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.514+0000 7f63ab0ea640 1 -- 172.21.15.46:0/2933868824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63a4101890 msgr2=0x7f63a4198be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:04.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.514+0000 7f63ab0ea640 1 --2- 172.21.15.46:0/2933868824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63a4101890 0x7f63a4198be0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6398006010 tx=0x7f63980042f0 comp rx=0 tx=0).stop 2024-04-26T21:32:04.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.514+0000 7f63ab0ea640 1 -- 172.21.15.46:0/2933868824 shutdown_connections 2024-04-26T21:32:04.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.514+0000 7f63ab0ea640 1 --2- 172.21.15.46:0/2933868824 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f63940a3c30 0x7f63940a60f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:04.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.514+0000 7f63ab0ea640 1 --2- 172.21.15.46:0/2933868824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63a4103440 0x7f63a41a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:04.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.514+0000 7f63ab0ea640 1 --2- 172.21.15.46:0/2933868824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f63a4102a80 0x7f63a4199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:04.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.514+0000 7f63ab0ea640 1 --2- 172.21.15.46:0/2933868824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63a4101890 0x7f63a4198be0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:04.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.514+0000 7f63ab0ea640 1 -- 172.21.15.46:0/2933868824 >> 172.21.15.46:0/2933868824 conn(0x7f63a40fce20 msgr2=0x7f63a4105ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:04.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.514+0000 7f63ab0ea640 1 -- 172.21.15.46:0/2933868824 shutdown_connections 2024-04-26T21:32:04.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.514+0000 7f63ab0ea640 1 -- 172.21.15.46:0/2933868824 wait complete. 2024-04-26T21:32:04.527 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T21:27:57.935988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T21:32:04.527 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T21:32:04.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.630+0000 7f11efcd1640 1 -- 172.21.15.46:0/3968864048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11e8101890 msgr2=0x7f11e8101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:04.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.630+0000 7f11efcd1640 1 --2- 172.21.15.46:0/3968864048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11e8101890 0x7f11e8101cb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f11e0009900 tx=0x7f11e002a410 comp rx=0 tx=0).stop 2024-04-26T21:32:04.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.630+0000 7f11efcd1640 1 -- 172.21.15.46:0/3968864048 shutdown_connections 2024-04-26T21:32:04.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.630+0000 7f11efcd1640 1 --2- 172.21.15.46:0/3968864048 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11e8103440 0x7f11e8109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:04.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.630+0000 7f11efcd1640 1 --2- 172.21.15.46:0/3968864048 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11e8102a80 0x7f11e8102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:04.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.630+0000 7f11efcd1640 1 --2- 172.21.15.46:0/3968864048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11e8101890 0x7f11e8101cb0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:04.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.630+0000 7f11efcd1640 1 -- 172.21.15.46:0/3968864048 >> 172.21.15.46:0/3968864048 conn(0x7f11e80fce00 msgr2=0x7f11e80ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:04.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.630+0000 7f11efcd1640 1 -- 172.21.15.46:0/3968864048 shutdown_connections 2024-04-26T21:32:04.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.630+0000 7f11efcd1640 1 -- 172.21.15.46:0/3968864048 wait complete. 2024-04-26T21:32:04.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.631+0000 7f11efcd1640 1 Processor -- start 2024-04-26T21:32:04.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.631+0000 7f11efcd1640 1 -- start start 2024-04-26T21:32:04.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.631+0000 7f11efcd1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11e8101890 0x7f11e8198cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:04.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.631+0000 7f11efcd1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11e8102a80 0x7f11e8199200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:04.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.631+0000 7f11efcd1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11e8103440 0x7f11e81a0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:04.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.631+0000 7f11efcd1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f11e810e540 con 0x7f11e8101890 2024-04-26T21:32:04.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.631+0000 7f11efcd1640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f11e810e3c0 con 0x7f11e8103440 2024-04-26T21:32:04.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.631+0000 7f11efcd1640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f11e810e6c0 con 0x7f11e8102a80 2024-04-26T21:32:04.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.631+0000 7f11eda6d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11e8101890 0x7f11e8198cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:04.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.631+0000 7f11eda6d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11e8101890 0x7f11e8198cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43576/0 (socket says 172.21.15.46:43576) 2024-04-26T21:32:04.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.631+0000 7f11eda6d640 1 -- 172.21.15.46:0/962258571 learned_addr learned my addr 172.21.15.46:0/962258571 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:04.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.631+0000 7f11ed26c640 1 --2- 172.21.15.46:0/962258571 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11e8102a80 0x7f11e8199200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:04.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.631+0000 7f11ee26e640 1 --2- 172.21.15.46:0/962258571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11e8103440 0x7f11e81a0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.632+0000 7f11eda6d640 1 -- 172.21.15.46:0/962258571 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11e8102a80 msgr2=0x7f11e8199200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.632+0000 7f11eda6d640 1 --2- 172.21.15.46:0/962258571 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11e8102a80 0x7f11e8199200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.632+0000 7f11eda6d640 1 -- 172.21.15.46:0/962258571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11e8103440 msgr2=0x7f11e81a0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.632+0000 7f11eda6d640 1 --2- 172.21.15.46:0/962258571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11e8103440 0x7f11e81a0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.632+0000 7f11eda6d640 1 -- 172.21.15.46:0/962258571 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11e81a0900 con 0x7f11e8101890 2024-04-26T21:32:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.632+0000 7f11ed26c640 1 --2- 172.21.15.46:0/962258571 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11e8102a80 0x7f11e8199200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:32:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.632+0000 7f11eda6d640 1 --2- 172.21.15.46:0/962258571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11e8101890 0x7f11e8198cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f11e0006010 tx=0x7f11e00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.632+0000 7f11deffd640 1 -- 172.21.15.46:0/962258571 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e003f030 con 0x7f11e8101890 2024-04-26T21:32:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.632+0000 7f11deffd640 1 -- 172.21.15.46:0/962258571 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11e00318f0 con 0x7f11e8101890 2024-04-26T21:32:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.632+0000 7f11efcd1640 1 -- 172.21.15.46:0/962258571 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11e81a0b90 con 0x7f11e8101890 2024-04-26T21:32:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.632+0000 7f11deffd640 1 -- 172.21.15.46:0/962258571 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e0035070 con 0x7f11e8101890 2024-04-26T21:32:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.632+0000 7f11efcd1640 1 -- 172.21.15.46:0/962258571 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11e81a1080 con 0x7f11e8101890 2024-04-26T21:32:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.633+0000 7f11efcd1640 1 -- 172.21.15.46:0/962258571 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11e8108cd0 con 0x7f11e8101890 2024-04-26T21:32:04.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.634+0000 7f11deffd640 1 -- 172.21.15.46:0/962258571 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f11e0042030 con 0x7f11e8101890 2024-04-26T21:32:04.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.635+0000 7f11deffd640 1 --2- 172.21.15.46:0/962258571 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f11d40a3bb0 0x7f11d40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:04.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.635+0000 7f11ed26c640 1 --2- 172.21.15.46:0/962258571 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f11d40a3bb0 0x7f11d40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:04.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.635+0000 7f11deffd640 1 -- 172.21.15.46:0/962258571 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f11e00f0630 con 0x7f11e8101890 2024-04-26T21:32:04.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.635+0000 7f11ed26c640 1 --2- 172.21.15.46:0/962258571 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f11d40a3bb0 0x7f11d40a6070 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f11e819a1c0 tx=0x7f11d8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:04.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.639+0000 7f11deffd640 1 -- 172.21.15.46:0/962258571 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f11e00bd410 con 0x7f11e8101890 2024-04-26T21:32:04.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:04 smithi046 ceph-mon[29201]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi050=b;smithi050=e;smithi050=h;smithi050=k;smithi142=c;smithi142=f;smithi142=i;smithi142=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:32:04.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:04 smithi046 ceph-mon[29201]: Saving service mds.all spec with placement smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi050=b;smithi050=e;smithi050=h;smithi050=k;smithi142=c;smithi142=f;smithi142=i;smithi142=l;count:12 2024-04-26T21:32:04.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:04 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:04.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:04 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:32:04.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:04 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:04.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:04 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:04.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:04 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:04.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:04 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:04.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:04 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2933868824' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:32:04.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.830+0000 7f11efcd1640 1 -- 172.21.15.46:0/962258571 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f11e802dad0 con 0x7f11e8101890 2024-04-26T21:32:04.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.830+0000 7f11deffd640 1 -- 172.21.15.46:0/962258571 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f11e00bcd10 con 0x7f11e8101890 2024-04-26T21:32:04.831 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-26T21:32:04.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.832+0000 7f11efcd1640 1 -- 172.21.15.46:0/962258571 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f11d40a3bb0 msgr2=0x7f11d40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:04.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.832+0000 7f11efcd1640 1 --2- 172.21.15.46:0/962258571 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f11d40a3bb0 0x7f11d40a6070 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f11e819a1c0 tx=0x7f11d8005f90 comp rx=0 tx=0).stop 2024-04-26T21:32:04.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.832+0000 7f11efcd1640 1 -- 172.21.15.46:0/962258571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11e8101890 msgr2=0x7f11e8198cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:04.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.832+0000 7f11efcd1640 1 --2- 172.21.15.46:0/962258571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11e8101890 0x7f11e8198cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f11e0006010 tx=0x7f11e00042f0 comp rx=0 tx=0).stop 2024-04-26T21:32:04.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.832+0000 7f11efcd1640 1 -- 172.21.15.46:0/962258571 shutdown_connections 2024-04-26T21:32:04.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.832+0000 7f11efcd1640 1 --2- 172.21.15.46:0/962258571 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f11d40a3bb0 0x7f11d40a6070 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:04.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.832+0000 7f11efcd1640 1 --2- 172.21.15.46:0/962258571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f11e8103440 0x7f11e81a0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:04.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.832+0000 7f11efcd1640 1 --2- 172.21.15.46:0/962258571 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f11e8102a80 0x7f11e8199200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:04.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.832+0000 7f11efcd1640 1 --2- 172.21.15.46:0/962258571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11e8101890 0x7f11e8198cc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:04.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.832+0000 7f11efcd1640 1 -- 172.21.15.46:0/962258571 >> 172.21.15.46:0/962258571 conn(0x7f11e80fce00 msgr2=0x7f11e8105ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:04.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.832+0000 7f11efcd1640 1 -- 172.21.15.46:0/962258571 shutdown_connections 2024-04-26T21:32:04.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.832+0000 7f11efcd1640 1 -- 172.21.15.46:0/962258571 wait complete. 2024-04-26T21:32:04.844 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:32:04.883 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:04 smithi142 ceph-mon[32944]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi050=b;smithi050=e;smithi050=h;smithi050=k;smithi142=c;smithi142=f;smithi142=i;smithi142=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:32:04.883 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:04 smithi142 ceph-mon[32944]: Saving service mds.all spec with placement smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi050=b;smithi050=e;smithi050=h;smithi050=k;smithi142=c;smithi142=f;smithi142=i;smithi142=l;count:12 2024-04-26T21:32:04.883 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:04 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:04.883 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:04 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:32:04.883 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:04 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:04.883 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:04 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:04.883 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:04 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:04.883 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:04 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:04.883 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:04 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/2933868824' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:32:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.945+0000 7f66b7359640 1 -- 172.21.15.46:0/2815435035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66b0105d80 msgr2=0x7f66b0108170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.945+0000 7f66b7359640 1 --2- 172.21.15.46:0/2815435035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66b0105d80 0x7f66b0108170 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f66a0009900 tx=0x7f66a002a410 comp rx=0 tx=0).stop 2024-04-26T21:32:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.945+0000 7f66b7359640 1 -- 172.21.15.46:0/2815435035 shutdown_connections 2024-04-26T21:32:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.945+0000 7f66b7359640 1 --2- 172.21.15.46:0/2815435035 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66b01086b0 0x7f66b010aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.945+0000 7f66b7359640 1 --2- 172.21.15.46:0/2815435035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66b0105d80 0x7f66b0108170 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.945+0000 7f66b7359640 1 --2- 172.21.15.46:0/2815435035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66b0103450 0x7f66b0105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.945+0000 7f66b7359640 1 -- 172.21.15.46:0/2815435035 >> 172.21.15.46:0/2815435035 conn(0x7f66b00fcde0 msgr2=0x7f66b00ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.945+0000 7f66b7359640 1 -- 172.21.15.46:0/2815435035 shutdown_connections 2024-04-26T21:32:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.945+0000 7f66b7359640 1 -- 172.21.15.46:0/2815435035 wait complete. 2024-04-26T21:32:04.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.945+0000 7f66b7359640 1 Processor -- start 2024-04-26T21:32:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.945+0000 7f66b7359640 1 -- start start 2024-04-26T21:32:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.945+0000 7f66b7359640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66b0103450 0x7f66b0102dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.946+0000 7f66b7359640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66b0105d80 0x7f66b0101400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.946+0000 7f66b7359640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66b01086b0 0x7f66b0101940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.946+0000 7f66b7359640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f66b010e470 con 0x7f66b0103450 2024-04-26T21:32:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.946+0000 7f66b7359640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f66b010e2f0 con 0x7f66b01086b0 2024-04-26T21:32:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.946+0000 7f66b7359640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f66b010e5f0 con 0x7f66b0105d80 2024-04-26T21:32:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.946+0000 7f66b50f5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66b0103450 0x7f66b0102dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.946+0000 7f66b50f5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66b0103450 0x7f66b0102dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43588/0 (socket says 172.21.15.46:43588) 2024-04-26T21:32:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.946+0000 7f66b50f5640 1 -- 172.21.15.46:0/968777969 learned_addr learned my addr 172.21.15.46:0/968777969 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.946+0000 7f66b48f4640 1 --2- 172.21.15.46:0/968777969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66b0105d80 0x7f66b0101400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.946+0000 7f66b58f6640 1 --2- 172.21.15.46:0/968777969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66b01086b0 0x7f66b0101940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.946+0000 7f66b50f5640 1 -- 172.21.15.46:0/968777969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66b0105d80 msgr2=0x7f66b0101400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.946+0000 7f66b50f5640 1 --2- 172.21.15.46:0/968777969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66b0105d80 0x7f66b0101400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.946+0000 7f66b50f5640 1 -- 172.21.15.46:0/968777969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66b01086b0 msgr2=0x7f66b0101940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.946+0000 7f66b50f5640 1 --2- 172.21.15.46:0/968777969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66b01086b0 0x7f66b0101940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:04.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.946+0000 7f66b50f5640 1 -- 172.21.15.46:0/968777969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b01021d0 con 0x7f66b0103450 2024-04-26T21:32:04.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.946+0000 7f66b58f6640 1 --2- 172.21.15.46:0/968777969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66b01086b0 0x7f66b0101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:32:04.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.946+0000 7f66b50f5640 1 --2- 172.21.15.46:0/968777969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66b0103450 0x7f66b0102dc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f66ac00d950 tx=0x7f66ac00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:04.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.946+0000 7f66a67fc640 1 -- 172.21.15.46:0/968777969 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66ac017070 con 0x7f66b0103450 2024-04-26T21:32:04.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.946+0000 7f66a67fc640 1 -- 172.21.15.46:0/968777969 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66ac009c30 con 0x7f66b0103450 2024-04-26T21:32:04.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.946+0000 7f66b7359640 1 -- 172.21.15.46:0/968777969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66b01a2d30 con 0x7f66b0103450 2024-04-26T21:32:04.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.946+0000 7f66a67fc640 1 -- 172.21.15.46:0/968777969 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66ac00f620 con 0x7f66b0103450 2024-04-26T21:32:04.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.946+0000 7f66b7359640 1 -- 172.21.15.46:0/968777969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66b01a31a0 con 0x7f66b0103450 2024-04-26T21:32:04.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.947+0000 7f66b7359640 1 -- 172.21.15.46:0/968777969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66b00fe8d0 con 0x7f66b0103450 2024-04-26T21:32:04.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.949+0000 7f66a67fc640 1 -- 172.21.15.46:0/968777969 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f66ac010460 con 0x7f66b0103450 2024-04-26T21:32:04.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.949+0000 7f66a67fc640 1 --2- 172.21.15.46:0/968777969 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f669c0a3bb0 0x7f669c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:04.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.949+0000 7f66a67fc640 1 -- 172.21.15.46:0/968777969 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f66ac0cfb90 con 0x7f66b0103450 2024-04-26T21:32:04.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.949+0000 7f66b48f4640 1 --2- 172.21.15.46:0/968777969 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f669c0a3bb0 0x7f669c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:04.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.950+0000 7f66b48f4640 1 --2- 172.21.15.46:0/968777969 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f669c0a3bb0 0x7f669c0a6070 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f66a0009230 tx=0x7f66a00314a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:04.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:04.954+0000 7f66a67fc640 1 -- 172.21.15.46:0/968777969 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f66ac09ca10 con 0x7f66b0103450 2024-04-26T21:32:05.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:04 smithi050 ceph-mon[32753]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi050=b;smithi050=e;smithi050=h;smithi050=k;smithi142=c;smithi142=f;smithi142=i;smithi142=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:32:05.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:04 smithi050 ceph-mon[32753]: Saving service mds.all spec with placement smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi050=b;smithi050=e;smithi050=h;smithi050=k;smithi142=c;smithi142=f;smithi142=i;smithi142=l;count:12 2024-04-26T21:32:05.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:04 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:05.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:04 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:32:05.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:04 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:05.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:04 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:05.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:04 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:05.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:04 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:05.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:04 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/2933868824' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:32:05.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:05.180+0000 7f66b7359640 1 -- 172.21.15.46:0/968777969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66b002dd40 con 0x7f66b0103450 2024-04-26T21:32:05.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:05.180+0000 7f66a67fc640 1 -- 172.21.15.46:0/968777969 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f66ac009de0 con 0x7f66b0103450 2024-04-26T21:32:05.181 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:32:05.181 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-26T21:32:05.181 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 1 2024-04-26T21:32:05.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:05.182+0000 7f66b7359640 1 -- 172.21.15.46:0/968777969 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f669c0a3bb0 msgr2=0x7f669c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:05.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:05.182+0000 7f66b7359640 1 --2- 172.21.15.46:0/968777969 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f669c0a3bb0 0x7f669c0a6070 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f66a0009230 tx=0x7f66a00314a0 comp rx=0 tx=0).stop 2024-04-26T21:32:05.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:05.182+0000 7f66b7359640 1 -- 172.21.15.46:0/968777969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66b0103450 msgr2=0x7f66b0102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:05.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:05.182+0000 7f66b7359640 1 --2- 172.21.15.46:0/968777969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66b0103450 0x7f66b0102dc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f66ac00d950 tx=0x7f66ac00de10 comp rx=0 tx=0).stop 2024-04-26T21:32:05.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:05.182+0000 7f66b7359640 1 -- 172.21.15.46:0/968777969 shutdown_connections 2024-04-26T21:32:05.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:05.182+0000 7f66b7359640 1 --2- 172.21.15.46:0/968777969 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f669c0a3bb0 0x7f669c0a6070 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:05.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:05.182+0000 7f66b7359640 1 --2- 172.21.15.46:0/968777969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66b01086b0 0x7f66b0101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:05.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:05.182+0000 7f66b7359640 1 --2- 172.21.15.46:0/968777969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f66b0105d80 0x7f66b0101400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:05.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:05.182+0000 7f66b7359640 1 --2- 172.21.15.46:0/968777969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66b0103450 0x7f66b0102dc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:05.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:05.182+0000 7f66b7359640 1 -- 172.21.15.46:0/968777969 >> 172.21.15.46:0/968777969 conn(0x7f66b00fcde0 msgr2=0x7f66b0103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:05.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:05.182+0000 7f66b7359640 1 -- 172.21.15.46:0/968777969 shutdown_connections 2024-04-26T21:32:05.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:05.182+0000 7f66b7359640 1 -- 172.21.15.46:0/968777969 wait complete. 2024-04-26T21:32:05.948 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:05 smithi142 ceph-mon[32944]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:05.949 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:05 smithi142 ceph-mon[32944]: Deploying daemon mds.l on smithi142 2024-04-26T21:32:05.949 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:05 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/962258571' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:32:05.949 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:05 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/968777969' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:06.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:05 smithi046 ceph-mon[29201]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:06.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:05 smithi046 ceph-mon[29201]: Deploying daemon mds.l on smithi142 2024-04-26T21:32:06.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:05 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/962258571' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:32:06.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:05 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/968777969' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:06.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:05 smithi050 ceph-mon[32753]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:06.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:05 smithi050 ceph-mon[32753]: Deploying daemon mds.l on smithi142 2024-04-26T21:32:06.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:05 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/962258571' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:32:06.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:05 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/968777969' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:06.290 INFO:journalctl@ceph.mds.l.smithi142.stdout:Apr 26 21:32:06 smithi142 systemd[1]: Starting Ceph mds.l for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:32:06.618 INFO:journalctl@ceph.mds.l.smithi142.stdout:Apr 26 21:32:06 smithi142 podman[63244]: 2024-04-26 21:32:06.237232453 +0000 UTC m=+0.017960193 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:32:06.618 INFO:journalctl@ceph.mds.l.smithi142.stdout:Apr 26 21:32:06 smithi142 podman[63244]: 2024-04-26 21:32:06.365054822 +0000 UTC m=+0.145782595 container create de319134d9918817af28adf90c2e8fb065658755b910192868787131f6d6cb3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-l, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:32:06.618 INFO:journalctl@ceph.mds.l.smithi142.stdout:Apr 26 21:32:06 smithi142 podman[63244]: 2024-04-26 21:32:06.617932279 +0000 UTC m=+0.398660024 container init de319134d9918817af28adf90c2e8fb065658755b910192868787131f6d6cb3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-l, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T21:32:07.041 INFO:journalctl@ceph.mds.l.smithi142.stdout:Apr 26 21:32:06 smithi142 podman[63244]: 2024-04-26 21:32:06.62061401 +0000 UTC m=+0.401341748 container start de319134d9918817af28adf90c2e8fb065658755b910192868787131f6d6cb3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424) 2024-04-26T21:32:07.041 INFO:journalctl@ceph.mds.l.smithi142.stdout:Apr 26 21:32:06 smithi142 ceph-mds[63263]: -- 172.21.15.142:0/3679055326 <== mon.1 v2:172.21.15.142:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5558eddd1a40 con 0x5558edda4c00 2024-04-26T21:32:07.041 INFO:journalctl@ceph.mds.l.smithi142.stdout:Apr 26 21:32:06 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-l[63259]: starting mds.l at 2024-04-26T21:32:07.041 INFO:journalctl@ceph.mds.l.smithi142.stdout:Apr 26 21:32:06 smithi142 bash[63244]: de319134d9918817af28adf90c2e8fb065658755b910192868787131f6d6cb3f 2024-04-26T21:32:07.041 INFO:journalctl@ceph.mds.l.smithi142.stdout:Apr 26 21:32:06 smithi142 systemd[1]: Started Ceph mds.l for 36c7013a-0413-11ef-bc93-c7b262605968. 2024-04-26T21:32:07.195 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:32:07.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.309+0000 7f6d2c971640 1 -- 172.21.15.46:0/1648589171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d28103440 msgr2=0x7f6d28109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.309+0000 7f6d2c971640 1 --2- 172.21.15.46:0/1648589171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d28103440 0x7f6d28109cd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6d20009ca0 tx=0x7f6d2002a8c0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.309+0000 7f6d2c971640 1 -- 172.21.15.46:0/1648589171 shutdown_connections 2024-04-26T21:32:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.309+0000 7f6d2c971640 1 --2- 172.21.15.46:0/1648589171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d28103440 0x7f6d28109cd0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.309+0000 7f6d2c971640 1 --2- 172.21.15.46:0/1648589171 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6d28102a80 0x7f6d28102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.309+0000 7f6d2c971640 1 --2- 172.21.15.46:0/1648589171 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d28101890 0x7f6d28101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.309+0000 7f6d2c971640 1 -- 172.21.15.46:0/1648589171 >> 172.21.15.46:0/1648589171 conn(0x7f6d280fce20 msgr2=0x7f6d280ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.309+0000 7f6d2c971640 1 -- 172.21.15.46:0/1648589171 shutdown_connections 2024-04-26T21:32:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.309+0000 7f6d2c971640 1 -- 172.21.15.46:0/1648589171 wait complete. 2024-04-26T21:32:07.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.310+0000 7f6d2c971640 1 Processor -- start 2024-04-26T21:32:07.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.310+0000 7f6d2c971640 1 -- start start 2024-04-26T21:32:07.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.310+0000 7f6d2c971640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d28101890 0x7f6d28105820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:07.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.310+0000 7f6d2c971640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6d28102a80 0x7f6d28103e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:07.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.310+0000 7f6d2c971640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d28103440 0x7f6d281043a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.310+0000 7f6d2c971640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2810d3a0 con 0x7f6d28103440 2024-04-26T21:32:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.310+0000 7f6d2c971640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2810d220 con 0x7f6d28101890 2024-04-26T21:32:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.310+0000 7f6d2c971640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2810d520 con 0x7f6d28102a80 2024-04-26T21:32:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.310+0000 7f6d26d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d28103440 0x7f6d281043a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.310+0000 7f6d26d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d28103440 0x7f6d281043a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47308/0 (socket says 172.21.15.46:47308) 2024-04-26T21:32:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.310+0000 7f6d26d9d640 1 -- 172.21.15.46:0/401720746 learned_addr learned my addr 172.21.15.46:0/401720746 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.310+0000 7f6d2659c640 1 --2- 172.21.15.46:0/401720746 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d28101890 0x7f6d28105820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.311+0000 7f6d1ffff640 1 --2- 172.21.15.46:0/401720746 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6d28102a80 0x7f6d28103e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.311+0000 7f6d26d9d640 1 -- 172.21.15.46:0/401720746 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6d28102a80 msgr2=0x7f6d28103e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.311+0000 7f6d26d9d640 1 --2- 172.21.15.46:0/401720746 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6d28102a80 0x7f6d28103e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.311+0000 7f6d26d9d640 1 -- 172.21.15.46:0/401720746 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d28101890 msgr2=0x7f6d28105820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.311+0000 7f6d26d9d640 1 --2- 172.21.15.46:0/401720746 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d28101890 0x7f6d28105820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.311+0000 7f6d26d9d640 1 -- 172.21.15.46:0/401720746 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d28104c30 con 0x7f6d28103440 2024-04-26T21:32:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.311+0000 7f6d1ffff640 1 --2- 172.21.15.46:0/401720746 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6d28102a80 0x7f6d28103e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:32:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.311+0000 7f6d26d9d640 1 --2- 172.21.15.46:0/401720746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d28103440 0x7f6d281043a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6d200097b0 tx=0x7f6d20031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.311+0000 7f6d1f7fe640 1 -- 172.21.15.46:0/401720746 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d20003bf0 con 0x7f6d28103440 2024-04-26T21:32:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.311+0000 7f6d2c971640 1 -- 172.21.15.46:0/401720746 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d281a2c60 con 0x7f6d28103440 2024-04-26T21:32:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.311+0000 7f6d1f7fe640 1 -- 172.21.15.46:0/401720746 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d20003d90 con 0x7f6d28103440 2024-04-26T21:32:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.311+0000 7f6d1f7fe640 1 -- 172.21.15.46:0/401720746 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d2002fc70 con 0x7f6d28103440 2024-04-26T21:32:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.311+0000 7f6d2c971640 1 -- 172.21.15.46:0/401720746 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d281a31b0 con 0x7f6d28103440 2024-04-26T21:32:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.312+0000 7f6d2c971640 1 -- 172.21.15.46:0/401720746 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d281a3380 con 0x7f6d28103440 2024-04-26T21:32:07.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.313+0000 7f6d1f7fe640 1 -- 172.21.15.46:0/401720746 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6d20004040 con 0x7f6d28103440 2024-04-26T21:32:07.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.314+0000 7f6d1f7fe640 1 --2- 172.21.15.46:0/401720746 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6d140a3bb0 0x7f6d140a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:07.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.314+0000 7f6d1f7fe640 1 -- 172.21.15.46:0/401720746 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6d200f15a0 con 0x7f6d28103440 2024-04-26T21:32:07.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.314+0000 7f6d2659c640 1 --2- 172.21.15.46:0/401720746 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6d140a3bb0 0x7f6d140a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:07.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.314+0000 7f6d2659c640 1 --2- 172.21.15.46:0/401720746 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6d140a3bb0 0x7f6d140a6070 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6d281028e0 tx=0x7f6d18005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:07.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.319+0000 7f6d1f7fe640 1 -- 172.21.15.46:0/401720746 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6d200be420 con 0x7f6d28103440 2024-04-26T21:32:07.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.549+0000 7f6d2c971640 1 -- 172.21.15.46:0/401720746 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d281a3670 con 0x7f6d28103440 2024-04-26T21:32:07.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.550+0000 7f6d1f7fe640 1 -- 172.21.15.46:0/401720746 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f6d200bdd20 con 0x7f6d28103440 2024-04-26T21:32:07.552 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:32:07.552 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3621399933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3621399933},{"type":"v1","addr":"172.21.15.142:6833","nonce":3621399933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T21:32:07.553 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 2 2024-04-26T21:32:07.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.552+0000 7f6d2c971640 1 -- 172.21.15.46:0/401720746 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6d140a3bb0 msgr2=0x7f6d140a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:07.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.552+0000 7f6d2c971640 1 --2- 172.21.15.46:0/401720746 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6d140a3bb0 0x7f6d140a6070 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6d281028e0 tx=0x7f6d18005f90 comp rx=0 tx=0).stop 2024-04-26T21:32:07.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.552+0000 7f6d2c971640 1 -- 172.21.15.46:0/401720746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d28103440 msgr2=0x7f6d281043a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:07.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.552+0000 7f6d2c971640 1 --2- 172.21.15.46:0/401720746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d28103440 0x7f6d281043a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6d200097b0 tx=0x7f6d20031840 comp rx=0 tx=0).stop 2024-04-26T21:32:07.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.552+0000 7f6d2c971640 1 -- 172.21.15.46:0/401720746 shutdown_connections 2024-04-26T21:32:07.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.552+0000 7f6d2c971640 1 --2- 172.21.15.46:0/401720746 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6d140a3bb0 0x7f6d140a6070 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.552+0000 7f6d2c971640 1 --2- 172.21.15.46:0/401720746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d28103440 0x7f6d281043a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.552+0000 7f6d2c971640 1 --2- 172.21.15.46:0/401720746 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6d28102a80 0x7f6d28103e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.552+0000 7f6d2c971640 1 --2- 172.21.15.46:0/401720746 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d28101890 0x7f6d28105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.552+0000 7f6d2c971640 1 -- 172.21.15.46:0/401720746 >> 172.21.15.46:0/401720746 conn(0x7f6d280fce20 msgr2=0x7f6d280fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:07.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.552+0000 7f6d2c971640 1 -- 172.21.15.46:0/401720746 shutdown_connections 2024-04-26T21:32:07.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:07.552+0000 7f6d2c971640 1 -- 172.21.15.46:0/401720746 wait complete. 2024-04-26T21:32:07.694 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:07 smithi050 ceph-mon[32753]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:07.694 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:07 smithi050 ceph-mon[32753]: mds.? [v2:172.21.15.142:6832/3621399933,v1:172.21.15.142:6833/3621399933] up:boot 2024-04-26T21:32:07.694 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:07 smithi050 ceph-mon[32753]: fsmap 1 up:standby 2024-04-26T21:32:07.694 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:07 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T21:32:07.694 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:07 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:07.694 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:07 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:07.694 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:07 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:07.694 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:07 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:07.694 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:07 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:07.694 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:07 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:07.694 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:07 smithi050 ceph-mon[32753]: Deploying daemon mds.k on smithi050 2024-04-26T21:32:07.695 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:07 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/401720746' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:08.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:07 smithi142 ceph-mon[32944]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:08.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:07 smithi142 ceph-mon[32944]: mds.? [v2:172.21.15.142:6832/3621399933,v1:172.21.15.142:6833/3621399933] up:boot 2024-04-26T21:32:08.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:07 smithi142 ceph-mon[32944]: fsmap 1 up:standby 2024-04-26T21:32:08.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:07 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T21:32:08.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:07 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:08.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:07 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:08.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:07 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:08.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:07 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:08.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:07 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:08.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:07 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:08.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:07 smithi142 ceph-mon[32944]: Deploying daemon mds.k on smithi050 2024-04-26T21:32:08.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:07 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/401720746' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:08.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:07 smithi046 ceph-mon[29201]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:08.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:07 smithi046 ceph-mon[29201]: mds.? [v2:172.21.15.142:6832/3621399933,v1:172.21.15.142:6833/3621399933] up:boot 2024-04-26T21:32:08.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:07 smithi046 ceph-mon[29201]: fsmap 1 up:standby 2024-04-26T21:32:08.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:07 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T21:32:08.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:07 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:08.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:07 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:08.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:07 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:08.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:07 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:08.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:07 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:08.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:07 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:08.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:07 smithi046 ceph-mon[29201]: Deploying daemon mds.k on smithi050 2024-04-26T21:32:08.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:07 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/401720746' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:09.311 INFO:journalctl@ceph.mds.k.smithi050.stdout:Apr 26 21:32:09 smithi050 systemd[1]: Starting Ceph mds.k for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:32:09.566 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:32:09.618 INFO:journalctl@ceph.mds.k.smithi050.stdout:Apr 26 21:32:09 smithi050 podman[62094]: 2024-04-26 21:32:09.213699955 +0000 UTC m=+0.018444882 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:32:09.618 INFO:journalctl@ceph.mds.k.smithi050.stdout:Apr 26 21:32:09 smithi050 podman[62094]: 2024-04-26 21:32:09.325812203 +0000 UTC m=+0.130557110 container create 07c58350ace453e95076c10bdf87c5589591dfc0b76322b588b173db9e231cfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-k, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-26T21:32:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.681+0000 7f557bd04640 1 -- 172.21.15.46:0/3337577226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5574103410 msgr2=0x7f5574105800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.681+0000 7f557bd04640 1 --2- 172.21.15.46:0/3337577226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5574103410 0x7f5574105800 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5570009900 tx=0x7f557002a410 comp rx=0 tx=0).stop 2024-04-26T21:32:09.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.681+0000 7f557bd04640 1 -- 172.21.15.46:0/3337577226 shutdown_connections 2024-04-26T21:32:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.681+0000 7f557bd04640 1 --2- 172.21.15.46:0/3337577226 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5574108670 0x7f557410aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.681+0000 7f557bd04640 1 --2- 172.21.15.46:0/3337577226 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5574105d40 0x7f5574108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.681+0000 7f557bd04640 1 --2- 172.21.15.46:0/3337577226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5574103410 0x7f5574105800 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.681+0000 7f557bd04640 1 -- 172.21.15.46:0/3337577226 >> 172.21.15.46:0/3337577226 conn(0x7f55740fce20 msgr2=0x7f55740ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.681+0000 7f557bd04640 1 -- 172.21.15.46:0/3337577226 shutdown_connections 2024-04-26T21:32:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.681+0000 7f557bd04640 1 -- 172.21.15.46:0/3337577226 wait complete. 2024-04-26T21:32:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.681+0000 7f557bd04640 1 Processor -- start 2024-04-26T21:32:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.682+0000 7f557bd04640 1 -- start start 2024-04-26T21:32:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.682+0000 7f557bd04640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5574103410 0x7f5574198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.682+0000 7f557bd04640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5574105d40 0x7f5574199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.682+0000 7f557bd04640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5574108670 0x7f55741a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.682+0000 7f557bd04640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f557410e4c0 con 0x7f5574108670 2024-04-26T21:32:09.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.682+0000 7f557bd04640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f557410e340 con 0x7f5574103410 2024-04-26T21:32:09.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.682+0000 7f557bd04640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f557410e640 con 0x7f5574105d40 2024-04-26T21:32:09.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.682+0000 7f557a2a1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5574108670 0x7f55741a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:09.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.682+0000 7f557a2a1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5574108670 0x7f55741a01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47326/0 (socket says 172.21.15.46:47326) 2024-04-26T21:32:09.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.682+0000 7f557a2a1640 1 -- 172.21.15.46:0/2534275395 learned_addr learned my addr 172.21.15.46:0/2534275395 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:09.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.682+0000 7f5579aa0640 1 --2- 172.21.15.46:0/2534275395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5574103410 0x7f5574198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:09.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.682+0000 7f557929f640 1 --2- 172.21.15.46:0/2534275395 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5574105d40 0x7f5574199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:09.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.682+0000 7f557a2a1640 1 -- 172.21.15.46:0/2534275395 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5574105d40 msgr2=0x7f5574199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:09.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.682+0000 7f557a2a1640 1 --2- 172.21.15.46:0/2534275395 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5574105d40 0x7f5574199190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:09.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.682+0000 7f557a2a1640 1 -- 172.21.15.46:0/2534275395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5574103410 msgr2=0x7f5574198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:09.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.682+0000 7f557a2a1640 1 --2- 172.21.15.46:0/2534275395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5574103410 0x7f5574198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:09.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.682+0000 7f557a2a1640 1 -- 172.21.15.46:0/2534275395 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55741a0860 con 0x7f5574108670 2024-04-26T21:32:09.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.682+0000 7f557a2a1640 1 --2- 172.21.15.46:0/2534275395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5574108670 0x7f55741a01c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f556c00eaa0 tx=0x7f556c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:09.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.683+0000 7f556affd640 1 -- 172.21.15.46:0/2534275395 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c00cdb0 con 0x7f5574108670 2024-04-26T21:32:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.683+0000 7f556affd640 1 -- 172.21.15.46:0/2534275395 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f556c005020 con 0x7f5574108670 2024-04-26T21:32:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.683+0000 7f557bd04640 1 -- 172.21.15.46:0/2534275395 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55741a0b50 con 0x7f5574108670 2024-04-26T21:32:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.683+0000 7f556affd640 1 -- 172.21.15.46:0/2534275395 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c0116d0 con 0x7f5574108670 2024-04-26T21:32:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.683+0000 7f557bd04640 1 -- 172.21.15.46:0/2534275395 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55741a1050 con 0x7f5574108670 2024-04-26T21:32:09.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.684+0000 7f557bd04640 1 -- 172.21.15.46:0/2534275395 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55740fe910 con 0x7f5574108670 2024-04-26T21:32:09.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.685+0000 7f556affd640 1 -- 172.21.15.46:0/2534275395 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f556c0051d0 con 0x7f5574108670 2024-04-26T21:32:09.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.685+0000 7f556affd640 1 --2- 172.21.15.46:0/2534275395 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f55600a3bb0 0x7f55600a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:09.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.685+0000 7f5579aa0640 1 --2- 172.21.15.46:0/2534275395 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f55600a3bb0 0x7f55600a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:09.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.686+0000 7f556affd640 1 -- 172.21.15.46:0/2534275395 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f556c0d0140 con 0x7f5574108670 2024-04-26T21:32:09.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.686+0000 7f5579aa0640 1 --2- 172.21.15.46:0/2534275395 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f55600a3bb0 0x7f55600a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5570009230 tx=0x7f55700314a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:09.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.690+0000 7f556affd640 1 -- 172.21.15.46:0/2534275395 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f556c016030 con 0x7f5574108670 2024-04-26T21:32:09.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.921+0000 7f557bd04640 1 -- 172.21.15.46:0/2534275395 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f557402dd20 con 0x7f5574108670 2024-04-26T21:32:09.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.921+0000 7f556affd640 1 -- 172.21.15.46:0/2534275395 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f556c09c970 con 0x7f5574108670 2024-04-26T21:32:09.922 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:32:09.922 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3621399933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3621399933},{"type":"v1","addr":"172.21.15.142:6833","nonce":3621399933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T21:32:09.923 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 2 2024-04-26T21:32:09.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.923+0000 7f557bd04640 1 -- 172.21.15.46:0/2534275395 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f55600a3bb0 msgr2=0x7f55600a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:09.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.923+0000 7f557bd04640 1 --2- 172.21.15.46:0/2534275395 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f55600a3bb0 0x7f55600a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5570009230 tx=0x7f55700314a0 comp rx=0 tx=0).stop 2024-04-26T21:32:09.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.923+0000 7f557bd04640 1 -- 172.21.15.46:0/2534275395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5574108670 msgr2=0x7f55741a01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:09.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.923+0000 7f557bd04640 1 --2- 172.21.15.46:0/2534275395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5574108670 0x7f55741a01c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f556c00eaa0 tx=0x7f556c00ef60 comp rx=0 tx=0).stop 2024-04-26T21:32:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.923+0000 7f557bd04640 1 -- 172.21.15.46:0/2534275395 shutdown_connections 2024-04-26T21:32:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.923+0000 7f557bd04640 1 --2- 172.21.15.46:0/2534275395 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f55600a3bb0 0x7f55600a6070 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.923+0000 7f557bd04640 1 --2- 172.21.15.46:0/2534275395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5574108670 0x7f55741a01c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.923+0000 7f557bd04640 1 --2- 172.21.15.46:0/2534275395 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5574105d40 0x7f5574199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.923+0000 7f557bd04640 1 --2- 172.21.15.46:0/2534275395 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5574103410 0x7f5574198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.923+0000 7f557bd04640 1 -- 172.21.15.46:0/2534275395 >> 172.21.15.46:0/2534275395 conn(0x7f55740fce20 msgr2=0x7f5574103e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.923+0000 7f557bd04640 1 -- 172.21.15.46:0/2534275395 shutdown_connections 2024-04-26T21:32:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:09.923+0000 7f557bd04640 1 -- 172.21.15.46:0/2534275395 wait complete. 2024-04-26T21:32:09.934 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:09 smithi046 ceph-mon[29201]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:10.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:09 smithi142 ceph-mon[32944]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:10.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:09 smithi050 ceph-mon[32753]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:10.118 INFO:journalctl@ceph.mds.k.smithi050.stdout:Apr 26 21:32:09 smithi050 podman[62094]: 2024-04-26 21:32:09.687114477 +0000 UTC m=+0.491859412 container init 07c58350ace453e95076c10bdf87c5589591dfc0b76322b588b173db9e231cfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-k, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-26T21:32:10.119 INFO:journalctl@ceph.mds.k.smithi050.stdout:Apr 26 21:32:09 smithi050 podman[62094]: 2024-04-26 21:32:09.690180836 +0000 UTC m=+0.494925755 container start 07c58350ace453e95076c10bdf87c5589591dfc0b76322b588b173db9e231cfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-04-26T21:32:10.119 INFO:journalctl@ceph.mds.k.smithi050.stdout:Apr 26 21:32:09 smithi050 ceph-mds[62113]: -- 172.21.15.50:0/1087765830 <== mon.2 v2:172.21.15.50:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562cd99ada40 con 0x562cd9980800 2024-04-26T21:32:10.119 INFO:journalctl@ceph.mds.k.smithi050.stdout:Apr 26 21:32:09 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-k[62109]: starting mds.k at 2024-04-26T21:32:10.119 INFO:journalctl@ceph.mds.k.smithi050.stdout:Apr 26 21:32:09 smithi050 bash[62094]: 07c58350ace453e95076c10bdf87c5589591dfc0b76322b588b173db9e231cfa 2024-04-26T21:32:10.119 INFO:journalctl@ceph.mds.k.smithi050.stdout:Apr 26 21:32:09 smithi050 systemd[1]: Started Ceph mds.k for 36c7013a-0413-11ef-bc93-c7b262605968. 2024-04-26T21:32:11.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:10 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:11.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:10 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:11.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:10 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:11.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:10 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:11.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:10 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:11.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:10 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:11.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:10 smithi050 ceph-mon[32753]: Deploying daemon mds.j on smithi046 2024-04-26T21:32:11.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:10 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/2534275395' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:11.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:10 smithi050 ceph-mon[32753]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:11.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:10 smithi050 ceph-mon[32753]: mds.? [v2:172.21.15.50:6832/345015778,v1:172.21.15.50:6833/345015778] up:boot 2024-04-26T21:32:11.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:10 smithi050 ceph-mon[32753]: fsmap 2 up:standby 2024-04-26T21:32:11.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:10 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T21:32:11.275 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:10 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:11.275 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:10 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:11.275 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:10 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:11.275 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:10 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:11.275 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:10 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:11.275 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:10 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:11.275 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:10 smithi046 ceph-mon[29201]: Deploying daemon mds.j on smithi046 2024-04-26T21:32:11.275 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:10 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2534275395' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:11.276 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:10 smithi046 ceph-mon[29201]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:11.276 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:10 smithi046 ceph-mon[29201]: mds.? [v2:172.21.15.50:6832/345015778,v1:172.21.15.50:6833/345015778] up:boot 2024-04-26T21:32:11.276 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:10 smithi046 ceph-mon[29201]: fsmap 2 up:standby 2024-04-26T21:32:11.276 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:10 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T21:32:11.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:10 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:11.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:10 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:11.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:10 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:11.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:10 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:11.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:10 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:11.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:10 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:11.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:10 smithi142 ceph-mon[32944]: Deploying daemon mds.j on smithi046 2024-04-26T21:32:11.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:10 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/2534275395' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:11.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:10 smithi142 ceph-mon[32944]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:11.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:10 smithi142 ceph-mon[32944]: mds.? [v2:172.21.15.50:6832/345015778,v1:172.21.15.50:6833/345015778] up:boot 2024-04-26T21:32:11.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:10 smithi142 ceph-mon[32944]: fsmap 2 up:standby 2024-04-26T21:32:11.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:10 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T21:32:11.936 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:32:12.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.049+0000 7f6609cb8640 1 -- 172.21.15.46:0/2618808844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6604103420 msgr2=0x7f6604109cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:12.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.049+0000 7f6609cb8640 1 --2- 172.21.15.46:0/2618808844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6604103420 0x7f6604109cb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f65fc009ca0 tx=0x7f65fc02a8c0 comp rx=0 tx=0).stop 2024-04-26T21:32:12.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.049+0000 7f6609cb8640 1 -- 172.21.15.46:0/2618808844 shutdown_connections 2024-04-26T21:32:12.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.049+0000 7f6609cb8640 1 --2- 172.21.15.46:0/2618808844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6604103420 0x7f6604109cb0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:12.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.049+0000 7f6609cb8640 1 --2- 172.21.15.46:0/2618808844 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6604102a60 0x7f6604102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:12.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.049+0000 7f6609cb8640 1 --2- 172.21.15.46:0/2618808844 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6604101870 0x7f6604101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:12.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.049+0000 7f6609cb8640 1 -- 172.21.15.46:0/2618808844 >> 172.21.15.46:0/2618808844 conn(0x7f66040fcde0 msgr2=0x7f66040ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:12.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.049+0000 7f6609cb8640 1 -- 172.21.15.46:0/2618808844 shutdown_connections 2024-04-26T21:32:12.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.049+0000 7f6609cb8640 1 -- 172.21.15.46:0/2618808844 wait complete. 2024-04-26T21:32:12.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.049+0000 7f6609cb8640 1 Processor -- start 2024-04-26T21:32:12.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.049+0000 7f6609cb8640 1 -- start start 2024-04-26T21:32:12.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.050+0000 7f6609cb8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6604101870 0x7f6604198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:12.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.050+0000 7f6609cb8640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6604102a60 0x7f6604199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:12.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.050+0000 7f6609cb8640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6604103420 0x7f66041a0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:12.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.050+0000 7f6609cb8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f660410d400 con 0x7f6604101870 2024-04-26T21:32:12.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.050+0000 7f6609cb8640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f660410d280 con 0x7f6604103420 2024-04-26T21:32:12.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.050+0000 7f6609cb8640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f660410d580 con 0x7f6604102a60 2024-04-26T21:32:12.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.050+0000 7f66037fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6604101870 0x7f6604198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:12.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.050+0000 7f66037fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6604101870 0x7f6604198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47346/0 (socket says 172.21.15.46:47346) 2024-04-26T21:32:12.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.050+0000 7f66037fe640 1 -- 172.21.15.46:0/3922619121 learned_addr learned my addr 172.21.15.46:0/3922619121 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:12.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.050+0000 7f6603fff640 1 --2- 172.21.15.46:0/3922619121 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6604103420 0x7f66041a0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:12.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.050+0000 7f65faffd640 1 --2- 172.21.15.46:0/3922619121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6604102a60 0x7f6604199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:12.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.050+0000 7f66037fe640 1 -- 172.21.15.46:0/3922619121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6604102a60 msgr2=0x7f6604199100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:12.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.050+0000 7f66037fe640 1 --2- 172.21.15.46:0/3922619121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6604102a60 0x7f6604199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:12.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.050+0000 7f66037fe640 1 -- 172.21.15.46:0/3922619121 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6604103420 msgr2=0x7f66041a0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:12.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.050+0000 7f66037fe640 1 --2- 172.21.15.46:0/3922619121 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6604103420 0x7f66041a0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:12.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.050+0000 7f66037fe640 1 -- 172.21.15.46:0/3922619121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66041a07e0 con 0x7f6604101870 2024-04-26T21:32:12.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.050+0000 7f66037fe640 1 --2- 172.21.15.46:0/3922619121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6604101870 0x7f6604198bc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f65f400e9c0 tx=0x7f65f400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:12.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.050+0000 7f66017fa640 1 -- 172.21.15.46:0/3922619121 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f400cd80 con 0x7f6604101870 2024-04-26T21:32:12.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.050+0000 7f6609cb8640 1 -- 172.21.15.46:0/3922619121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66041a0ad0 con 0x7f6604101870 2024-04-26T21:32:12.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.050+0000 7f66017fa640 1 -- 172.21.15.46:0/3922619121 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65f40035f0 con 0x7f6604101870 2024-04-26T21:32:12.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.050+0000 7f66017fa640 1 -- 172.21.15.46:0/3922619121 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f4010690 con 0x7f6604101870 2024-04-26T21:32:12.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.050+0000 7f6609cb8640 1 -- 172.21.15.46:0/3922619121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66041a1020 con 0x7f6604101870 2024-04-26T21:32:12.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.051+0000 7f6609cb8640 1 -- 172.21.15.46:0/3922619121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66041a1250 con 0x7f6604101870 2024-04-26T21:32:12.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.053+0000 7f66017fa640 1 -- 172.21.15.46:0/3922619121 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f65f40037a0 con 0x7f6604101870 2024-04-26T21:32:12.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.053+0000 7f66017fa640 1 --2- 172.21.15.46:0/3922619121 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f65f00a3bb0 0x7f65f00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:12.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.054+0000 7f65faffd640 1 --2- 172.21.15.46:0/3922619121 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f65f00a3bb0 0x7f65f00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:12.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.054+0000 7f66017fa640 1 -- 172.21.15.46:0/3922619121 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f65f40d04b0 con 0x7f6604101870 2024-04-26T21:32:12.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.054+0000 7f65faffd640 1 --2- 172.21.15.46:0/3922619121 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f65f00a3bb0 0x7f65f00a6070 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f660419a0d0 tx=0x7f65ec008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:12.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.057+0000 7f66017fa640 1 -- 172.21.15.46:0/3922619121 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f65f409d310 con 0x7f6604101870 2024-04-26T21:32:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.295+0000 7f6609cb8640 1 -- 172.21.15.46:0/3922619121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66041a1590 con 0x7f6604101870 2024-04-26T21:32:12.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.295+0000 7f66017fa640 1 -- 172.21.15.46:0/3922619121 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2218 (secure 0 0 0) 0x7f65f409cc10 con 0x7f6604101870 2024-04-26T21:32:12.296 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:32:12.296 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3621399933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3621399933},{"type":"v1","addr":"172.21.15.142:6833","nonce":3621399933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/345015778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":345015778},{"type":"v1","addr":"172.21.15.50:6833","nonce":345015778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-26T21:32:12.297 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 3 2024-04-26T21:32:12.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.297+0000 7f6609cb8640 1 -- 172.21.15.46:0/3922619121 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f65f00a3bb0 msgr2=0x7f65f00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:12.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.297+0000 7f6609cb8640 1 --2- 172.21.15.46:0/3922619121 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f65f00a3bb0 0x7f65f00a6070 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f660419a0d0 tx=0x7f65ec008040 comp rx=0 tx=0).stop 2024-04-26T21:32:12.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.297+0000 7f6609cb8640 1 -- 172.21.15.46:0/3922619121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6604101870 msgr2=0x7f6604198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:12.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.297+0000 7f6609cb8640 1 --2- 172.21.15.46:0/3922619121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6604101870 0x7f6604198bc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f65f400e9c0 tx=0x7f65f400ee80 comp rx=0 tx=0).stop 2024-04-26T21:32:12.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.297+0000 7f6609cb8640 1 -- 172.21.15.46:0/3922619121 shutdown_connections 2024-04-26T21:32:12.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.297+0000 7f6609cb8640 1 --2- 172.21.15.46:0/3922619121 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f65f00a3bb0 0x7f65f00a6070 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:12.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.297+0000 7f6609cb8640 1 --2- 172.21.15.46:0/3922619121 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6604103420 0x7f66041a0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:12.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.297+0000 7f6609cb8640 1 --2- 172.21.15.46:0/3922619121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6604102a60 0x7f6604199100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:12.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.297+0000 7f6609cb8640 1 --2- 172.21.15.46:0/3922619121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6604101870 0x7f6604198bc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:12.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.297+0000 7f6609cb8640 1 -- 172.21.15.46:0/3922619121 >> 172.21.15.46:0/3922619121 conn(0x7f66040fcde0 msgr2=0x7f66040fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:12.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.297+0000 7f6609cb8640 1 -- 172.21.15.46:0/3922619121 shutdown_connections 2024-04-26T21:32:12.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:12.297+0000 7f6609cb8640 1 -- 172.21.15.46:0/3922619121 wait complete. 2024-04-26T21:32:12.303 INFO:journalctl@ceph.mds.j.smithi046.stdout:Apr 26 21:32:12 smithi046 systemd[1]: Starting Ceph mds.j for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:32:12.577 INFO:journalctl@ceph.mds.j.smithi046.stdout:Apr 26 21:32:12 smithi046 podman[69275]: 2024-04-26 21:32:12.286399308 +0000 UTC m=+0.022954686 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:32:12.577 INFO:journalctl@ceph.mds.j.smithi046.stdout:Apr 26 21:32:12 smithi046 podman[69275]: 2024-04-26 21:32:12.428937705 +0000 UTC m=+0.165493071 container create a38069bfbd621c83a40728775bea4129db3882e1d8b404031acdc0c0ee630d7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-j, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True) 2024-04-26T21:32:12.980 INFO:journalctl@ceph.mds.j.smithi046.stdout:Apr 26 21:32:12 smithi046 podman[69275]: 2024-04-26 21:32:12.69841193 +0000 UTC m=+0.434967299 container init a38069bfbd621c83a40728775bea4129db3882e1d8b404031acdc0c0ee630d7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True) 2024-04-26T21:32:12.980 INFO:journalctl@ceph.mds.j.smithi046.stdout:Apr 26 21:32:12 smithi046 podman[69275]: 2024-04-26 21:32:12.701116232 +0000 UTC m=+0.437671608 container start a38069bfbd621c83a40728775bea4129db3882e1d8b404031acdc0c0ee630d7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-j, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-26T21:32:12.980 INFO:journalctl@ceph.mds.j.smithi046.stdout:Apr 26 21:32:12 smithi046 ceph-mds[69296]: -- 172.21.15.46:0/1185730064 <== mon.0 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556aae223a40 con 0x556aae1f6800 2024-04-26T21:32:12.980 INFO:journalctl@ceph.mds.j.smithi046.stdout:Apr 26 21:32:12 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-j[69292]: starting mds.j at 2024-04-26T21:32:12.980 INFO:journalctl@ceph.mds.j.smithi046.stdout:Apr 26 21:32:12 smithi046 bash[69275]: a38069bfbd621c83a40728775bea4129db3882e1d8b404031acdc0c0ee630d7d 2024-04-26T21:32:12.981 INFO:journalctl@ceph.mds.j.smithi046.stdout:Apr 26 21:32:12 smithi046 systemd[1]: Started Ceph mds.j for 36c7013a-0413-11ef-bc93-c7b262605968. 2024-04-26T21:32:13.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:12 smithi142 ceph-mon[32944]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:13.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:12 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3922619121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:13.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:12 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:13.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:12 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:13.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:12 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:13.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:12 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:13.292 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:12 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:13.292 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:12 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:13.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:12 smithi046 ceph-mon[29201]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:13.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:12 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3922619121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:13.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:12 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:13.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:12 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:13.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:12 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:13.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:12 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:13.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:12 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:13.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:12 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:13.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:12 smithi050 ceph-mon[32753]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:13.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:12 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3922619121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:13.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:12 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:13.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:12 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:13.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:12 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:13.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:12 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:13.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:12 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:13.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:12 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:14.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:13 smithi142 ceph-mon[32944]: Deploying daemon mds.d on smithi046 2024-04-26T21:32:14.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:13 smithi142 ceph-mon[32944]: mds.? [v2:172.21.15.46:6834/3674296286,v1:172.21.15.46:6835/3674296286] up:boot 2024-04-26T21:32:14.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:13 smithi142 ceph-mon[32944]: fsmap 3 up:standby 2024-04-26T21:32:14.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:13 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T21:32:14.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:13 smithi046 ceph-mon[29201]: Deploying daemon mds.d on smithi046 2024-04-26T21:32:14.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:13 smithi046 ceph-mon[29201]: mds.? [v2:172.21.15.46:6834/3674296286,v1:172.21.15.46:6835/3674296286] up:boot 2024-04-26T21:32:14.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:13 smithi046 ceph-mon[29201]: fsmap 3 up:standby 2024-04-26T21:32:14.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:13 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T21:32:14.310 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:32:14.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:13 smithi050 ceph-mon[32753]: Deploying daemon mds.d on smithi046 2024-04-26T21:32:14.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:13 smithi050 ceph-mon[32753]: mds.? [v2:172.21.15.46:6834/3674296286,v1:172.21.15.46:6835/3674296286] up:boot 2024-04-26T21:32:14.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:13 smithi050 ceph-mon[32753]: fsmap 3 up:standby 2024-04-26T21:32:14.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:13 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T21:32:14.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.420+0000 7fb5aae93640 1 -- 172.21.15.46:0/2401172889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5a4103420 msgr2=0x7fb5a4109cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:14.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.420+0000 7fb5aae93640 1 --2- 172.21.15.46:0/2401172889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5a4103420 0x7fb5a4109cb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb594009900 tx=0x7fb59402a410 comp rx=0 tx=0).stop 2024-04-26T21:32:14.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.420+0000 7fb5aae93640 1 -- 172.21.15.46:0/2401172889 shutdown_connections 2024-04-26T21:32:14.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.420+0000 7fb5aae93640 1 --2- 172.21.15.46:0/2401172889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5a4103420 0x7fb5a4109cb0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:14.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.420+0000 7fb5aae93640 1 --2- 172.21.15.46:0/2401172889 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5a4102a60 0x7fb5a4102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:14.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.420+0000 7fb5aae93640 1 --2- 172.21.15.46:0/2401172889 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5a4101870 0x7fb5a4101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:14.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.420+0000 7fb5aae93640 1 -- 172.21.15.46:0/2401172889 >> 172.21.15.46:0/2401172889 conn(0x7fb5a40fcde0 msgr2=0x7fb5a40ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:14.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.420+0000 7fb5aae93640 1 -- 172.21.15.46:0/2401172889 shutdown_connections 2024-04-26T21:32:14.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.420+0000 7fb5aae93640 1 -- 172.21.15.46:0/2401172889 wait complete. 2024-04-26T21:32:14.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.420+0000 7fb5aae93640 1 Processor -- start 2024-04-26T21:32:14.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.420+0000 7fb5aae93640 1 -- start start 2024-04-26T21:32:14.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.421+0000 7fb5aae93640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5a4101870 0x7fb5a4198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:14.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.421+0000 7fb5aae93640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5a4102a60 0x7fb5a4199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:14.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.421+0000 7fb5aae93640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5a4103420 0x7fb5a41a00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:14.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.421+0000 7fb5aae93640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a410e4c0 con 0x7fb5a4102a60 2024-04-26T21:32:14.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.421+0000 7fb5aae93640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a410e340 con 0x7fb5a4101870 2024-04-26T21:32:14.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.421+0000 7fb5aae93640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a410e640 con 0x7fb5a4103420 2024-04-26T21:32:14.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.421+0000 7fb5a3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5a4102a60 0x7fb5a4199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:14.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.421+0000 7fb5a3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5a4102a60 0x7fb5a4199060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47374/0 (socket says 172.21.15.46:47374) 2024-04-26T21:32:14.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.421+0000 7fb5a3fff640 1 -- 172.21.15.46:0/912516972 learned_addr learned my addr 172.21.15.46:0/912516972 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:14.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.421+0000 7fb5a9430640 1 --2- 172.21.15.46:0/912516972 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5a4103420 0x7fb5a41a00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:14.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.421+0000 7fb5a8c2f640 1 --2- 172.21.15.46:0/912516972 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5a4101870 0x7fb5a4198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:14.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.421+0000 7fb5a3fff640 1 -- 172.21.15.46:0/912516972 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5a4103420 msgr2=0x7fb5a41a00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:14.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.421+0000 7fb5a3fff640 1 --2- 172.21.15.46:0/912516972 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5a4103420 0x7fb5a41a00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:14.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.421+0000 7fb5a3fff640 1 -- 172.21.15.46:0/912516972 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5a4101870 msgr2=0x7fb5a4198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:14.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.421+0000 7fb5a3fff640 1 --2- 172.21.15.46:0/912516972 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5a4101870 0x7fb5a4198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:14.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.421+0000 7fb5a3fff640 1 -- 172.21.15.46:0/912516972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5a41a0740 con 0x7fb5a4102a60 2024-04-26T21:32:14.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.421+0000 7fb5a3fff640 1 --2- 172.21.15.46:0/912516972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5a4102a60 0x7fb5a4199060 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb59800c890 tx=0x7fb59800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:14.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.421+0000 7fb5a1ffb640 1 -- 172.21.15.46:0/912516972 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb598004220 con 0x7fb5a4102a60 2024-04-26T21:32:14.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.421+0000 7fb5aae93640 1 -- 172.21.15.46:0/912516972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5a41a0a30 con 0x7fb5a4102a60 2024-04-26T21:32:14.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.421+0000 7fb5a1ffb640 1 -- 172.21.15.46:0/912516972 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5980043c0 con 0x7fb5a4102a60 2024-04-26T21:32:14.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.421+0000 7fb5a1ffb640 1 -- 172.21.15.46:0/912516972 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb598004a00 con 0x7fb5a4102a60 2024-04-26T21:32:14.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.422+0000 7fb5aae93640 1 -- 172.21.15.46:0/912516972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5a41a0fd0 con 0x7fb5a4102a60 2024-04-26T21:32:14.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.423+0000 7fb5aae93640 1 -- 172.21.15.46:0/912516972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5a4108c10 con 0x7fb5a4102a60 2024-04-26T21:32:14.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.424+0000 7fb5a1ffb640 1 -- 172.21.15.46:0/912516972 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb5980079b0 con 0x7fb5a4102a60 2024-04-26T21:32:14.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.424+0000 7fb5a1ffb640 1 --2- 172.21.15.46:0/912516972 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb58c0a3bb0 0x7fb58c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:14.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.424+0000 7fb5a8c2f640 1 --2- 172.21.15.46:0/912516972 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb58c0a3bb0 0x7fb58c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:14.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.425+0000 7fb5a1ffb640 1 -- 172.21.15.46:0/912516972 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb5980cfd80 con 0x7fb5a4102a60 2024-04-26T21:32:14.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.425+0000 7fb5a8c2f640 1 --2- 172.21.15.46:0/912516972 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb58c0a3bb0 0x7fb58c0a6070 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb5a41028c0 tx=0x7fb59c009460 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:14.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.429+0000 7fb5a1ffb640 1 -- 172.21.15.46:0/912516972 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb598014030 con 0x7fb5a4102a60 2024-04-26T21:32:14.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.657+0000 7fb5aae93640 1 -- 172.21.15.46:0/912516972 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb5a4063530 con 0x7fb5a4102a60 2024-04-26T21:32:14.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.658+0000 7fb5a1ffb640 1 -- 172.21.15.46:0/912516972 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3043 (secure 0 0 0) 0x7fb59809cb60 con 0x7fb5a4102a60 2024-04-26T21:32:14.659 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:32:14.659 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3674296286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3674296286},{"type":"v1","addr":"172.21.15.46:6835","nonce":3674296286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3621399933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3621399933},{"type":"v1","addr":"172.21.15.142:6833","nonce":3621399933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/345015778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":345015778},{"type":"v1","addr":"172.21.15.50:6833","nonce":345015778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-26T21:32:14.659 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 4 2024-04-26T21:32:14.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.660+0000 7fb5aae93640 1 -- 172.21.15.46:0/912516972 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb58c0a3bb0 msgr2=0x7fb58c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:14.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.660+0000 7fb5aae93640 1 --2- 172.21.15.46:0/912516972 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb58c0a3bb0 0x7fb58c0a6070 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb5a41028c0 tx=0x7fb59c009460 comp rx=0 tx=0).stop 2024-04-26T21:32:14.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.660+0000 7fb5aae93640 1 -- 172.21.15.46:0/912516972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5a4102a60 msgr2=0x7fb5a4199060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:14.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.660+0000 7fb5aae93640 1 --2- 172.21.15.46:0/912516972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5a4102a60 0x7fb5a4199060 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb59800c890 tx=0x7fb59800cd50 comp rx=0 tx=0).stop 2024-04-26T21:32:14.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.660+0000 7fb5aae93640 1 -- 172.21.15.46:0/912516972 shutdown_connections 2024-04-26T21:32:14.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.660+0000 7fb5aae93640 1 --2- 172.21.15.46:0/912516972 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb58c0a3bb0 0x7fb58c0a6070 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:14.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.660+0000 7fb5aae93640 1 --2- 172.21.15.46:0/912516972 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb5a4103420 0x7fb5a41a00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:14.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.660+0000 7fb5aae93640 1 --2- 172.21.15.46:0/912516972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5a4102a60 0x7fb5a4199060 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:14.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.660+0000 7fb5aae93640 1 --2- 172.21.15.46:0/912516972 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5a4101870 0x7fb5a4198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:14.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.660+0000 7fb5aae93640 1 -- 172.21.15.46:0/912516972 >> 172.21.15.46:0/912516972 conn(0x7fb5a40fcde0 msgr2=0x7fb5a4105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:14.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.660+0000 7fb5aae93640 1 -- 172.21.15.46:0/912516972 shutdown_connections 2024-04-26T21:32:14.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:14.660+0000 7fb5aae93640 1 -- 172.21.15.46:0/912516972 wait complete. 2024-04-26T21:32:15.187 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:14 smithi046 ceph-mon[29201]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:15.187 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:14 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/912516972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:15.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:14 smithi142 ceph-mon[32944]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:15.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:14 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/912516972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:15.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:14 smithi050 ceph-mon[32753]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:15.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:14 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/912516972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:15.553 INFO:journalctl@ceph.mds.d.smithi046.stdout:Apr 26 21:32:15 smithi046 podman[69623]: 2024-04-26 21:32:15.436026545 +0000 UTC m=+0.145326487 container create 04bfdfb737279183a375a429450ac4d60d0309f68e45461702311c7a0b181f0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-26T21:32:16.054 INFO:journalctl@ceph.mds.d.smithi046.stdout:Apr 26 21:32:15 smithi046 podman[69623]: 2024-04-26 21:32:15.705858207 +0000 UTC m=+0.415158143 container init 04bfdfb737279183a375a429450ac4d60d0309f68e45461702311c7a0b181f0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-d, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-26T21:32:16.054 INFO:journalctl@ceph.mds.d.smithi046.stdout:Apr 26 21:32:15 smithi046 podman[69623]: 2024-04-26 21:32:15.708627122 +0000 UTC m=+0.417927061 container start 04bfdfb737279183a375a429450ac4d60d0309f68e45461702311c7a0b181f0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-26T21:32:16.054 INFO:journalctl@ceph.mds.d.smithi046.stdout:Apr 26 21:32:15 smithi046 ceph-mds[69642]: -- 172.21.15.46:0/194655270 <== mon.0 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5591879fba40 con 0x5591879cf000 2024-04-26T21:32:16.054 INFO:journalctl@ceph.mds.d.smithi046.stdout:Apr 26 21:32:15 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-d[69638]: starting mds.d at 2024-04-26T21:32:16.054 INFO:journalctl@ceph.mds.d.smithi046.stdout:Apr 26 21:32:15 smithi046 bash[69623]: 04bfdfb737279183a375a429450ac4d60d0309f68e45461702311c7a0b181f0f 2024-04-26T21:32:16.054 INFO:journalctl@ceph.mds.d.smithi046.stdout:Apr 26 21:32:15 smithi046 systemd[1]: Started Ceph mds.d for 36c7013a-0413-11ef-bc93-c7b262605968. 2024-04-26T21:32:16.673 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:32:16.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.790+0000 7f4c9c797640 1 -- 172.21.15.46:0/2420857732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c941033f0 msgr2=0x7f4c941057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:16.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.790+0000 7f4c9c797640 1 --2- 172.21.15.46:0/2420857732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c941033f0 0x7f4c941057e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c009900 tx=0x7f4c8c02a410 comp rx=0 tx=0).stop 2024-04-26T21:32:16.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.790+0000 7f4c9c797640 1 -- 172.21.15.46:0/2420857732 shutdown_connections 2024-04-26T21:32:16.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.790+0000 7f4c9c797640 1 --2- 172.21.15.46:0/2420857732 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c94108650 0x7f4c9410aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:16.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.790+0000 7f4c9c797640 1 --2- 172.21.15.46:0/2420857732 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c94105d20 0x7f4c94108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:16.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.790+0000 7f4c9c797640 1 --2- 172.21.15.46:0/2420857732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c941033f0 0x7f4c941057e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:16.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.790+0000 7f4c9c797640 1 -- 172.21.15.46:0/2420857732 >> 172.21.15.46:0/2420857732 conn(0x7f4c940fcde0 msgr2=0x7f4c940ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:16.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.790+0000 7f4c9c797640 1 -- 172.21.15.46:0/2420857732 shutdown_connections 2024-04-26T21:32:16.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.790+0000 7f4c9c797640 1 -- 172.21.15.46:0/2420857732 wait complete. 2024-04-26T21:32:16.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.791+0000 7f4c9c797640 1 Processor -- start 2024-04-26T21:32:16.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.791+0000 7f4c9c797640 1 -- start start 2024-04-26T21:32:16.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.791+0000 7f4c9c797640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c941033f0 0x7f4c94198d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:16.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.791+0000 7f4c9c797640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c94105d20 0x7f4c94199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:16.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.791+0000 7f4c9c797640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c94108650 0x7f4c941a02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:16.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.791+0000 7f4c9c797640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9410e590 con 0x7f4c94108650 2024-04-26T21:32:16.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.791+0000 7f4c9c797640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9410e410 con 0x7f4c94105d20 2024-04-26T21:32:16.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.791+0000 7f4c9c797640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9410e710 con 0x7f4c941033f0 2024-04-26T21:32:16.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.791+0000 7f4c9ad34640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c94108650 0x7f4c941a02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:16.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.791+0000 7f4c9ad34640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c94108650 0x7f4c941a02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40002/0 (socket says 172.21.15.46:40002) 2024-04-26T21:32:16.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.791+0000 7f4c9ad34640 1 -- 172.21.15.46:0/2715576528 learned_addr learned my addr 172.21.15.46:0/2715576528 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.791+0000 7f4c9a533640 1 --2- 172.21.15.46:0/2715576528 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c941033f0 0x7f4c94198d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.791+0000 7f4c99d32640 1 --2- 172.21.15.46:0/2715576528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c94105d20 0x7f4c94199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.791+0000 7f4c9ad34640 1 -- 172.21.15.46:0/2715576528 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c941033f0 msgr2=0x7f4c94198d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.791+0000 7f4c9ad34640 1 --2- 172.21.15.46:0/2715576528 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c941033f0 0x7f4c94198d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.791+0000 7f4c9ad34640 1 -- 172.21.15.46:0/2715576528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c94105d20 msgr2=0x7f4c94199290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.791+0000 7f4c9ad34640 1 --2- 172.21.15.46:0/2715576528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c94105d20 0x7f4c94199290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.791+0000 7f4c9ad34640 1 -- 172.21.15.46:0/2715576528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c941a0960 con 0x7f4c94108650 2024-04-26T21:32:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.791+0000 7f4c9ad34640 1 --2- 172.21.15.46:0/2715576528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c94108650 0x7f4c941a02c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4c8400b970 tx=0x7f4c8400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.792+0000 7f4c937fe640 1 -- 172.21.15.46:0/2715576528 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c84019070 con 0x7f4c94108650 2024-04-26T21:32:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.792+0000 7f4c937fe640 1 -- 172.21.15.46:0/2715576528 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c84014440 con 0x7f4c94108650 2024-04-26T21:32:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.792+0000 7f4c9c797640 1 -- 172.21.15.46:0/2715576528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c941a0c50 con 0x7f4c94108650 2024-04-26T21:32:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.792+0000 7f4c937fe640 1 -- 172.21.15.46:0/2715576528 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c840075d0 con 0x7f4c94108650 2024-04-26T21:32:16.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.792+0000 7f4c9c797640 1 -- 172.21.15.46:0/2715576528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c941a11a0 con 0x7f4c94108650 2024-04-26T21:32:16.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.793+0000 7f4c9c797640 1 -- 172.21.15.46:0/2715576528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c940fe8d0 con 0x7f4c94108650 2024-04-26T21:32:16.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.794+0000 7f4c937fe640 1 -- 172.21.15.46:0/2715576528 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4c8401c070 con 0x7f4c94108650 2024-04-26T21:32:16.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.794+0000 7f4c937fe640 1 --2- 172.21.15.46:0/2715576528 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4c7c0a3c30 0x7f4c7c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:16.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.795+0000 7f4c9a533640 1 --2- 172.21.15.46:0/2715576528 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4c7c0a3c30 0x7f4c7c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:16.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.795+0000 7f4c937fe640 1 -- 172.21.15.46:0/2715576528 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f4c840cf4c0 con 0x7f4c94108650 2024-04-26T21:32:16.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.795+0000 7f4c9a533640 1 --2- 172.21.15.46:0/2715576528 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4c7c0a3c30 0x7f4c7c0a60f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c009230 tx=0x7f4c8c0314a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:16.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:16.799+0000 7f4c937fe640 1 -- 172.21.15.46:0/2715576528 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4c8409bc50 con 0x7f4c94108650 2024-04-26T21:32:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:17.028+0000 7f4c9c797640 1 -- 172.21.15.46:0/2715576528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4c940635d0 con 0x7f4c94108650 2024-04-26T21:32:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:17.029+0000 7f4c937fe640 1 -- 172.21.15.46:0/2715576528 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3865 (secure 0 0 0) 0x7f4c8409ff50 con 0x7f4c94108650 2024-04-26T21:32:17.030 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:32:17.030 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3674296286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3674296286},{"type":"v1","addr":"172.21.15.46:6835","nonce":3674296286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/391791206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":391791206},{"type":"v1","addr":"172.21.15.46:6837","nonce":391791206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3621399933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3621399933},{"type":"v1","addr":"172.21.15.142:6833","nonce":3621399933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/345015778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":345015778},{"type":"v1","addr":"172.21.15.50:6833","nonce":345015778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-26T21:32:17.030 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 5 2024-04-26T21:32:17.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:17.031+0000 7f4c9c797640 1 -- 172.21.15.46:0/2715576528 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4c7c0a3c30 msgr2=0x7f4c7c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:17.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:17.031+0000 7f4c9c797640 1 --2- 172.21.15.46:0/2715576528 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4c7c0a3c30 0x7f4c7c0a60f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c009230 tx=0x7f4c8c0314a0 comp rx=0 tx=0).stop 2024-04-26T21:32:17.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:17.031+0000 7f4c9c797640 1 -- 172.21.15.46:0/2715576528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c94108650 msgr2=0x7f4c941a02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:17.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:17.031+0000 7f4c9c797640 1 --2- 172.21.15.46:0/2715576528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c94108650 0x7f4c941a02c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4c8400b970 tx=0x7f4c8400bd70 comp rx=0 tx=0).stop 2024-04-26T21:32:17.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:17.031+0000 7f4c9c797640 1 -- 172.21.15.46:0/2715576528 shutdown_connections 2024-04-26T21:32:17.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:17.031+0000 7f4c9c797640 1 --2- 172.21.15.46:0/2715576528 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4c7c0a3c30 0x7f4c7c0a60f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:17.031+0000 7f4c9c797640 1 --2- 172.21.15.46:0/2715576528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4c94108650 0x7f4c941a02c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:17.031+0000 7f4c9c797640 1 --2- 172.21.15.46:0/2715576528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4c94105d20 0x7f4c94199290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:17.031+0000 7f4c9c797640 1 --2- 172.21.15.46:0/2715576528 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4c941033f0 0x7f4c94198d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:17.031+0000 7f4c9c797640 1 -- 172.21.15.46:0/2715576528 >> 172.21.15.46:0/2715576528 conn(0x7f4c940fcde0 msgr2=0x7f4c94103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:17.031+0000 7f4c9c797640 1 -- 172.21.15.46:0/2715576528 shutdown_connections 2024-04-26T21:32:17.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:17.031+0000 7f4c9c797640 1 -- 172.21.15.46:0/2715576528 wait complete. 2024-04-26T21:32:17.042 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:16 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:17.042 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:16 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:17.043 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:16 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:17.043 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:16 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:17.043 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:16 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:17.043 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:16 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:17.043 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:16 smithi046 ceph-mon[29201]: Deploying daemon mds.e on smithi050 2024-04-26T21:32:17.043 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:16 smithi046 ceph-mon[29201]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:17.043 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:16 smithi046 ceph-mon[29201]: mds.? [v2:172.21.15.46:6836/391791206,v1:172.21.15.46:6837/391791206] up:boot 2024-04-26T21:32:17.043 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:16 smithi046 ceph-mon[29201]: fsmap 4 up:standby 2024-04-26T21:32:17.043 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:16 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T21:32:17.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:16 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:17.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:16 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:17.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:16 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:17.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:16 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:17.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:16 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:17.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:16 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:17.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:16 smithi050 ceph-mon[32753]: Deploying daemon mds.e on smithi050 2024-04-26T21:32:17.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:16 smithi050 ceph-mon[32753]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:17.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:16 smithi050 ceph-mon[32753]: mds.? [v2:172.21.15.46:6836/391791206,v1:172.21.15.46:6837/391791206] up:boot 2024-04-26T21:32:17.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:16 smithi050 ceph-mon[32753]: fsmap 4 up:standby 2024-04-26T21:32:17.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:16 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T21:32:17.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:16 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:17.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:16 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:17.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:16 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:17.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:16 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:17.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:16 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:17.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:16 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:17.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:16 smithi142 ceph-mon[32944]: Deploying daemon mds.e on smithi050 2024-04-26T21:32:17.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:16 smithi142 ceph-mon[32944]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:17.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:16 smithi142 ceph-mon[32944]: mds.? [v2:172.21.15.46:6836/391791206,v1:172.21.15.46:6837/391791206] up:boot 2024-04-26T21:32:17.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:16 smithi142 ceph-mon[32944]: fsmap 4 up:standby 2024-04-26T21:32:17.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:16 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T21:32:18.254 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:17 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/2715576528' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:18.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:17 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/2715576528' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:18.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:17 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2715576528' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:18.868 INFO:journalctl@ceph.mds.e.smithi050.stdout:Apr 26 21:32:18 smithi050 systemd[1]: Started Ceph mds.e for 36c7013a-0413-11ef-bc93-c7b262605968. 2024-04-26T21:32:19.044 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:32:19.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.157+0000 7f168461d640 1 -- 172.21.15.46:0/3037820469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f167c1024b0 msgr2=0x7f167c109d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:19.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.157+0000 7f168461d640 1 --2- 172.21.15.46:0/3037820469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f167c1024b0 0x7f167c109d50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1674009ca0 tx=0x7f167402a8c0 comp rx=0 tx=0).stop 2024-04-26T21:32:19.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.157+0000 7f168461d640 1 -- 172.21.15.46:0/3037820469 shutdown_connections 2024-04-26T21:32:19.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.157+0000 7f168461d640 1 --2- 172.21.15.46:0/3037820469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f167c1024b0 0x7f167c109d50 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:19.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.157+0000 7f168461d640 1 --2- 172.21.15.46:0/3037820469 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f167c101af0 0x7f167c101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:19.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.157+0000 7f168461d640 1 --2- 172.21.15.46:0/3037820469 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f167c102ea0 0x7f167c1032c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:19.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.157+0000 7f168461d640 1 -- 172.21.15.46:0/3037820469 >> 172.21.15.46:0/3037820469 conn(0x7f167c0fc5a0 msgr2=0x7f167c0fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:19.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.157+0000 7f168461d640 1 -- 172.21.15.46:0/3037820469 shutdown_connections 2024-04-26T21:32:19.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.157+0000 7f168461d640 1 -- 172.21.15.46:0/3037820469 wait complete. 2024-04-26T21:32:19.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.158+0000 7f168461d640 1 Processor -- start 2024-04-26T21:32:19.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.158+0000 7f168461d640 1 -- start start 2024-04-26T21:32:19.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.158+0000 7f168461d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f167c101af0 0x7f167c198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:19.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.158+0000 7f168461d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f167c1024b0 0x7f167c1990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:19.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.158+0000 7f168461d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f167c102ea0 0x7f167c1a0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:19.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.158+0000 7f168461d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f167c10e540 con 0x7f167c102ea0 2024-04-26T21:32:19.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.158+0000 7f168461d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f167c10e3c0 con 0x7f167c101af0 2024-04-26T21:32:19.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.158+0000 7f168461d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f167c10e6c0 con 0x7f167c1024b0 2024-04-26T21:32:19.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.158+0000 7f1682bba640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f167c102ea0 0x7f167c1a0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:19.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.158+0000 7f1682bba640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f167c102ea0 0x7f167c1a0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40022/0 (socket says 172.21.15.46:40022) 2024-04-26T21:32:19.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.158+0000 7f1682bba640 1 -- 172.21.15.46:0/1440827179 learned_addr learned my addr 172.21.15.46:0/1440827179 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:19.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.158+0000 7f1681bb8640 1 --2- 172.21.15.46:0/1440827179 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f167c1024b0 0x7f167c1990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:19.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.158+0000 7f16823b9640 1 --2- 172.21.15.46:0/1440827179 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f167c101af0 0x7f167c198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:19.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.159+0000 7f1682bba640 1 -- 172.21.15.46:0/1440827179 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f167c1024b0 msgr2=0x7f167c1990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:19.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.159+0000 7f1682bba640 1 --2- 172.21.15.46:0/1440827179 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f167c1024b0 0x7f167c1990d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:19.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.159+0000 7f1682bba640 1 -- 172.21.15.46:0/1440827179 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f167c101af0 msgr2=0x7f167c198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:19.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.159+0000 7f1682bba640 1 --2- 172.21.15.46:0/1440827179 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f167c101af0 0x7f167c198b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:19.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.159+0000 7f1682bba640 1 -- 172.21.15.46:0/1440827179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f167c1a07b0 con 0x7f167c102ea0 2024-04-26T21:32:19.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.159+0000 7f1682bba640 1 --2- 172.21.15.46:0/1440827179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f167c102ea0 0x7f167c1a0140 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f16740097b0 tx=0x7f167402f390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:19.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.159+0000 7f16737fe640 1 -- 172.21.15.46:0/1440827179 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1674003c70 con 0x7f167c102ea0 2024-04-26T21:32:19.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.159+0000 7f16737fe640 1 -- 172.21.15.46:0/1440827179 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1674003e10 con 0x7f167c102ea0 2024-04-26T21:32:19.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.159+0000 7f168461d640 1 -- 172.21.15.46:0/1440827179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f167c1a0a40 con 0x7f167c102ea0 2024-04-26T21:32:19.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.159+0000 7f16737fe640 1 -- 172.21.15.46:0/1440827179 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f167402fb90 con 0x7f167c102ea0 2024-04-26T21:32:19.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.159+0000 7f168461d640 1 -- 172.21.15.46:0/1440827179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f167c1a0f90 con 0x7f167c102ea0 2024-04-26T21:32:19.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.160+0000 7f168461d640 1 -- 172.21.15.46:0/1440827179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f167c108ca0 con 0x7f167c102ea0 2024-04-26T21:32:19.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.161+0000 7f16737fe640 1 -- 172.21.15.46:0/1440827179 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1674004680 con 0x7f167c102ea0 2024-04-26T21:32:19.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.162+0000 7f16737fe640 1 --2- 172.21.15.46:0/1440827179 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f16680a3bb0 0x7f16680a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:19.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.162+0000 7f16823b9640 1 --2- 172.21.15.46:0/1440827179 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f16680a3bb0 0x7f16680a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:19.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.162+0000 7f16737fe640 1 -- 172.21.15.46:0/1440827179 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f16740f1650 con 0x7f167c102ea0 2024-04-26T21:32:19.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.162+0000 7f16823b9640 1 --2- 172.21.15.46:0/1440827179 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f16680a3bb0 0x7f16680a6070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f167c06add0 tx=0x7f1678005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:19.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.166+0000 7f16737fe640 1 -- 172.21.15.46:0/1440827179 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f167402fd70 con 0x7f167c102ea0 2024-04-26T21:32:19.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:18 smithi142 ceph-mon[32944]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:19.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:18 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:19.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:18 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:19.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:18 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:19.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:18 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:19.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:18 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:19.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:18 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:19.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:18 smithi046 ceph-mon[29201]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:19.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:18 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:19.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:18 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:19.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:18 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:19.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:18 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:19.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:18 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:19.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:18 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:19.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:18 smithi050 ceph-mon[32753]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:19.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:18 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:19.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:18 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:19.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:18 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:19.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:18 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:19.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:18 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:19.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:18 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:19.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.391+0000 7f168461d640 1 -- 172.21.15.46:0/1440827179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f167c02dad0 con 0x7f167c102ea0 2024-04-26T21:32:19.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.391+0000 7f16737fe640 1 -- 172.21.15.46:0/1440827179 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4690 (secure 0 0 0) 0x7f16740be430 con 0x7f167c102ea0 2024-04-26T21:32:19.393 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:32:19.393 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3674296286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3674296286},{"type":"v1","addr":"172.21.15.46:6835","nonce":3674296286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/391791206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":391791206},{"type":"v1","addr":"172.21.15.46:6837","nonce":391791206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3621399933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3621399933},{"type":"v1","addr":"172.21.15.142:6833","nonce":3621399933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/345015778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":345015778},{"type":"v1","addr":"172.21.15.50:6833","nonce":345015778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/2546509863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":2546509863},{"type":"v1","addr":"172.21.15.50:6835","nonce":2546509863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-26T21:32:19.393 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 6 2024-04-26T21:32:19.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.393+0000 7f168461d640 1 -- 172.21.15.46:0/1440827179 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f16680a3bb0 msgr2=0x7f16680a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:19.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.393+0000 7f168461d640 1 --2- 172.21.15.46:0/1440827179 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f16680a3bb0 0x7f16680a6070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f167c06add0 tx=0x7f1678005f90 comp rx=0 tx=0).stop 2024-04-26T21:32:19.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.393+0000 7f168461d640 1 -- 172.21.15.46:0/1440827179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f167c102ea0 msgr2=0x7f167c1a0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:19.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.393+0000 7f168461d640 1 --2- 172.21.15.46:0/1440827179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f167c102ea0 0x7f167c1a0140 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f16740097b0 tx=0x7f167402f390 comp rx=0 tx=0).stop 2024-04-26T21:32:19.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.393+0000 7f168461d640 1 -- 172.21.15.46:0/1440827179 shutdown_connections 2024-04-26T21:32:19.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.393+0000 7f168461d640 1 --2- 172.21.15.46:0/1440827179 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f16680a3bb0 0x7f16680a6070 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:19.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.393+0000 7f168461d640 1 --2- 172.21.15.46:0/1440827179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f167c102ea0 0x7f167c1a0140 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:19.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.393+0000 7f168461d640 1 --2- 172.21.15.46:0/1440827179 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f167c1024b0 0x7f167c1990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:19.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.393+0000 7f168461d640 1 --2- 172.21.15.46:0/1440827179 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f167c101af0 0x7f167c198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:19.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.393+0000 7f168461d640 1 -- 172.21.15.46:0/1440827179 >> 172.21.15.46:0/1440827179 conn(0x7f167c0fc5a0 msgr2=0x7f167c105e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:19.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.393+0000 7f168461d640 1 -- 172.21.15.46:0/1440827179 shutdown_connections 2024-04-26T21:32:19.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:19.393+0000 7f168461d640 1 -- 172.21.15.46:0/1440827179 wait complete. 2024-04-26T21:32:20.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:19 smithi142 ceph-mon[32944]: Deploying daemon mds.f on smithi142 2024-04-26T21:32:20.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:19 smithi142 ceph-mon[32944]: mds.? [v2:172.21.15.50:6834/2546509863,v1:172.21.15.50:6835/2546509863] up:boot 2024-04-26T21:32:20.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:19 smithi142 ceph-mon[32944]: fsmap 5 up:standby 2024-04-26T21:32:20.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:19 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T21:32:20.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:19 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/1440827179' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:20.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:19 smithi046 ceph-mon[29201]: Deploying daemon mds.f on smithi142 2024-04-26T21:32:20.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:19 smithi046 ceph-mon[29201]: mds.? [v2:172.21.15.50:6834/2546509863,v1:172.21.15.50:6835/2546509863] up:boot 2024-04-26T21:32:20.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:19 smithi046 ceph-mon[29201]: fsmap 5 up:standby 2024-04-26T21:32:20.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:19 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T21:32:20.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:19 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1440827179' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:20.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:19 smithi050 ceph-mon[32753]: Deploying daemon mds.f on smithi142 2024-04-26T21:32:20.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:19 smithi050 ceph-mon[32753]: mds.? [v2:172.21.15.50:6834/2546509863,v1:172.21.15.50:6835/2546509863] up:boot 2024-04-26T21:32:20.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:19 smithi050 ceph-mon[32753]: fsmap 5 up:standby 2024-04-26T21:32:20.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:19 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T21:32:20.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:19 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/1440827179' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:21.194 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:20 smithi142 ceph-mon[32944]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:21.194 INFO:journalctl@ceph.mds.f.smithi142.stdout:Apr 26 21:32:20 smithi142 systemd[1]: Starting Ceph mds.f for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:32:21.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:20 smithi046 ceph-mon[29201]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:21.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:20 smithi050 ceph-mon[32753]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:21.406 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:32:21.496 INFO:journalctl@ceph.mds.f.smithi142.stdout:Apr 26 21:32:21 smithi142 podman[63553]: 2024-04-26 21:32:21.193326849 +0000 UTC m=+0.112136957 container create 2bf98b6eacb3c827841f375600cd8c06bc10a7cb8c711da5b9b2c9bb58b696b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-f, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-26T21:32:21.496 INFO:journalctl@ceph.mds.f.smithi142.stdout:Apr 26 21:32:21 smithi142 podman[63553]: 2024-04-26 21:32:21.099261213 +0000 UTC m=+0.018071291 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:32:21.496 INFO:journalctl@ceph.mds.f.smithi142.stdout:Apr 26 21:32:21 smithi142 podman[63553]: 2024-04-26 21:32:21.495953319 +0000 UTC m=+0.414763385 container init 2bf98b6eacb3c827841f375600cd8c06bc10a7cb8c711da5b9b2c9bb58b696b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-f, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T21:32:21.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.519+0000 7f59ce4d1640 1 -- 172.21.15.46:0/69769126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59c8101870 msgr2=0x7f59c8101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:21.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.519+0000 7f59ce4d1640 1 --2- 172.21.15.46:0/69769126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59c8101870 0x7f59c8101c90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f59b8009900 tx=0x7f59b802a410 comp rx=0 tx=0).stop 2024-04-26T21:32:21.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.520+0000 7f59ce4d1640 1 -- 172.21.15.46:0/69769126 shutdown_connections 2024-04-26T21:32:21.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.520+0000 7f59ce4d1640 1 --2- 172.21.15.46:0/69769126 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59c8103420 0x7f59c8109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:21.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.520+0000 7f59ce4d1640 1 --2- 172.21.15.46:0/69769126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59c8102a60 0x7f59c8102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:21.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.520+0000 7f59ce4d1640 1 --2- 172.21.15.46:0/69769126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59c8101870 0x7f59c8101c90 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:21.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.520+0000 7f59ce4d1640 1 -- 172.21.15.46:0/69769126 >> 172.21.15.46:0/69769126 conn(0x7f59c80fcde0 msgr2=0x7f59c80ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:21.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.520+0000 7f59ce4d1640 1 -- 172.21.15.46:0/69769126 shutdown_connections 2024-04-26T21:32:21.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.520+0000 7f59ce4d1640 1 -- 172.21.15.46:0/69769126 wait complete. 2024-04-26T21:32:21.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.520+0000 7f59ce4d1640 1 Processor -- start 2024-04-26T21:32:21.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.520+0000 7f59ce4d1640 1 -- start start 2024-04-26T21:32:21.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.521+0000 7f59ce4d1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59c8101870 0x7f59c8198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:21.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.521+0000 7f59ce4d1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59c8102a60 0x7f59c8199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:21.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.521+0000 7f59ce4d1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59c8103420 0x7f59c81a0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:21.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.521+0000 7f59ce4d1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f59c810d3f0 con 0x7f59c8102a60 2024-04-26T21:32:21.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.521+0000 7f59ce4d1640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f59c810d270 con 0x7f59c8101870 2024-04-26T21:32:21.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.521+0000 7f59ce4d1640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f59c810d570 con 0x7f59c8103420 2024-04-26T21:32:21.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.521+0000 7f59c77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59c8102a60 0x7f59c8199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:21.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.521+0000 7f59c77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59c8102a60 0x7f59c8199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40034/0 (socket says 172.21.15.46:40034) 2024-04-26T21:32:21.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.521+0000 7f59c77fe640 1 -- 172.21.15.46:0/852258820 learned_addr learned my addr 172.21.15.46:0/852258820 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:21.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.521+0000 7f59cca6e640 1 --2- 172.21.15.46:0/852258820 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59c8103420 0x7f59c81a0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:21.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.521+0000 7f59c7fff640 1 --2- 172.21.15.46:0/852258820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59c8101870 0x7f59c8198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:21.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.521+0000 7f59c77fe640 1 -- 172.21.15.46:0/852258820 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59c8103420 msgr2=0x7f59c81a0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:21.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.521+0000 7f59c77fe640 1 --2- 172.21.15.46:0/852258820 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59c8103420 0x7f59c81a0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:21.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.521+0000 7f59c77fe640 1 -- 172.21.15.46:0/852258820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59c8101870 msgr2=0x7f59c8198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:21.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.521+0000 7f59c77fe640 1 --2- 172.21.15.46:0/852258820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59c8101870 0x7f59c8198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:21.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.521+0000 7f59c77fe640 1 -- 172.21.15.46:0/852258820 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59c81a07e0 con 0x7f59c8102a60 2024-04-26T21:32:21.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.521+0000 7f59c77fe640 1 --2- 172.21.15.46:0/852258820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59c8102a60 0x7f59c8199100 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f59bc00e9f0 tx=0x7f59bc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:21.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.521+0000 7f59c57fa640 1 -- 172.21.15.46:0/852258820 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59bc00cd80 con 0x7f59c8102a60 2024-04-26T21:32:21.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.521+0000 7f59c57fa640 1 -- 172.21.15.46:0/852258820 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59bc0108e0 con 0x7f59c8102a60 2024-04-26T21:32:21.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.522+0000 7f59ce4d1640 1 -- 172.21.15.46:0/852258820 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59c81a0ad0 con 0x7f59c8102a60 2024-04-26T21:32:21.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.522+0000 7f59c57fa640 1 -- 172.21.15.46:0/852258820 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59bc018980 con 0x7f59c8102a60 2024-04-26T21:32:21.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.522+0000 7f59ce4d1640 1 -- 172.21.15.46:0/852258820 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59c81a1020 con 0x7f59c8102a60 2024-04-26T21:32:21.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.523+0000 7f59ce4d1640 1 -- 172.21.15.46:0/852258820 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59c81a11f0 con 0x7f59c8102a60 2024-04-26T21:32:21.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.524+0000 7f59c57fa640 1 -- 172.21.15.46:0/852258820 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f59bc010430 con 0x7f59c8102a60 2024-04-26T21:32:21.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.524+0000 7f59c57fa640 1 --2- 172.21.15.46:0/852258820 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f59b00a3bb0 0x7f59b00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:21.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.524+0000 7f59c7fff640 1 --2- 172.21.15.46:0/852258820 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f59b00a3bb0 0x7f59b00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:21.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.524+0000 7f59c57fa640 1 -- 172.21.15.46:0/852258820 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f59bc0d02c0 con 0x7f59c8102a60 2024-04-26T21:32:21.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.525+0000 7f59c7fff640 1 --2- 172.21.15.46:0/852258820 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f59b00a3bb0 0x7f59b00a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f59b8006010 tx=0x7f59b80314a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:21.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.529+0000 7f59c57fa640 1 -- 172.21.15.46:0/852258820 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f59bc09d140 con 0x7f59c8102a60 2024-04-26T21:32:21.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.756+0000 7f59ce4d1640 1 -- 172.21.15.46:0/852258820 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59c81a1580 con 0x7f59c8102a60 2024-04-26T21:32:21.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.756+0000 7f59c57fa640 1 -- 172.21.15.46:0/852258820 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4690 (secure 0 0 0) 0x7f59bc09ca40 con 0x7f59c8102a60 2024-04-26T21:32:21.757 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:32:21.758 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3674296286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3674296286},{"type":"v1","addr":"172.21.15.46:6835","nonce":3674296286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/391791206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":391791206},{"type":"v1","addr":"172.21.15.46:6837","nonce":391791206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3621399933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3621399933},{"type":"v1","addr":"172.21.15.142:6833","nonce":3621399933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/345015778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":345015778},{"type":"v1","addr":"172.21.15.50:6833","nonce":345015778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/2546509863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":2546509863},{"type":"v1","addr":"172.21.15.50:6835","nonce":2546509863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-26T21:32:21.758 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 6 2024-04-26T21:32:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.758+0000 7f59ce4d1640 1 -- 172.21.15.46:0/852258820 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f59b00a3bb0 msgr2=0x7f59b00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.758+0000 7f59ce4d1640 1 --2- 172.21.15.46:0/852258820 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f59b00a3bb0 0x7f59b00a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f59b8006010 tx=0x7f59b80314a0 comp rx=0 tx=0).stop 2024-04-26T21:32:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.758+0000 7f59ce4d1640 1 -- 172.21.15.46:0/852258820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59c8102a60 msgr2=0x7f59c8199100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.758+0000 7f59ce4d1640 1 --2- 172.21.15.46:0/852258820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59c8102a60 0x7f59c8199100 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f59bc00e9f0 tx=0x7f59bc00eeb0 comp rx=0 tx=0).stop 2024-04-26T21:32:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.759+0000 7f59ce4d1640 1 -- 172.21.15.46:0/852258820 shutdown_connections 2024-04-26T21:32:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.759+0000 7f59ce4d1640 1 --2- 172.21.15.46:0/852258820 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f59b00a3bb0 0x7f59b00a6070 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.759+0000 7f59ce4d1640 1 --2- 172.21.15.46:0/852258820 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f59c8103420 0x7f59c81a0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.759+0000 7f59ce4d1640 1 --2- 172.21.15.46:0/852258820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59c8102a60 0x7f59c8199100 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.759+0000 7f59ce4d1640 1 --2- 172.21.15.46:0/852258820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f59c8101870 0x7f59c8198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:21.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.759+0000 7f59ce4d1640 1 -- 172.21.15.46:0/852258820 >> 172.21.15.46:0/852258820 conn(0x7f59c80fcde0 msgr2=0x7f59c80fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:21.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.759+0000 7f59ce4d1640 1 -- 172.21.15.46:0/852258820 shutdown_connections 2024-04-26T21:32:21.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:21.759+0000 7f59ce4d1640 1 -- 172.21.15.46:0/852258820 wait complete. 2024-04-26T21:32:21.790 INFO:journalctl@ceph.mds.f.smithi142.stdout:Apr 26 21:32:21 smithi142 podman[63553]: 2024-04-26 21:32:21.498499802 +0000 UTC m=+0.417309877 container start 2bf98b6eacb3c827841f375600cd8c06bc10a7cb8c711da5b9b2c9bb58b696b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-f, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-26T21:32:21.791 INFO:journalctl@ceph.mds.f.smithi142.stdout:Apr 26 21:32:21 smithi142 ceph-mds[63572]: -- 172.21.15.142:0/3513384127 <== mon.1 v2:172.21.15.142:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561202779a40 con 0x56120274d000 2024-04-26T21:32:21.791 INFO:journalctl@ceph.mds.f.smithi142.stdout:Apr 26 21:32:21 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-f[63568]: starting mds.f at 2024-04-26T21:32:21.791 INFO:journalctl@ceph.mds.f.smithi142.stdout:Apr 26 21:32:21 smithi142 bash[63553]: 2bf98b6eacb3c827841f375600cd8c06bc10a7cb8c711da5b9b2c9bb58b696b3 2024-04-26T21:32:21.791 INFO:journalctl@ceph.mds.f.smithi142.stdout:Apr 26 21:32:21 smithi142 systemd[1]: Started Ceph mds.f for 36c7013a-0413-11ef-bc93-c7b262605968. 2024-04-26T21:32:22.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:22 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:22.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:22 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:22.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:22 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:22.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:22 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:22.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:22 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:22.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:22 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:22.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:22 smithi050 ceph-mon[32753]: Deploying daemon mds.c on smithi142 2024-04-26T21:32:22.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:22 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/852258820' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:22.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:22 smithi050 ceph-mon[32753]: mds.? [v2:172.21.15.142:6834/3791468066,v1:172.21.15.142:6835/3791468066] up:boot 2024-04-26T21:32:22.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:22 smithi050 ceph-mon[32753]: fsmap 6 up:standby 2024-04-26T21:32:22.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:22 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T21:32:23.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:22 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:23.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:22 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:23.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:22 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:23.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:22 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:23.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:22 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:23.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:22 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:23.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:22 smithi142 ceph-mon[32944]: Deploying daemon mds.c on smithi142 2024-04-26T21:32:23.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:22 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/852258820' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:23.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:22 smithi142 ceph-mon[32944]: mds.? [v2:172.21.15.142:6834/3791468066,v1:172.21.15.142:6835/3791468066] up:boot 2024-04-26T21:32:23.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:22 smithi142 ceph-mon[32944]: fsmap 6 up:standby 2024-04-26T21:32:23.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:22 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T21:32:23.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:22 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:23.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:22 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:23.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:22 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:23.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:22 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:23.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:22 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:23.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:22 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:23.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:22 smithi046 ceph-mon[29201]: Deploying daemon mds.c on smithi142 2024-04-26T21:32:23.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:22 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/852258820' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:23.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:22 smithi046 ceph-mon[29201]: mds.? [v2:172.21.15.142:6834/3791468066,v1:172.21.15.142:6835/3791468066] up:boot 2024-04-26T21:32:23.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:22 smithi046 ceph-mon[29201]: fsmap 6 up:standby 2024-04-26T21:32:23.055 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:22 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T21:32:23.771 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:32:23.935 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:23 smithi046 ceph-mon[29201]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:23.987 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:23 smithi142 ceph-mon[32944]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:23.988 INFO:journalctl@ceph.mds.c.smithi142.stdout:Apr 26 21:32:23 smithi142 systemd[1]: Starting Ceph mds.c for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:32:24.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.025+0000 7fb1198b8640 1 -- 172.21.15.46:0/1566518871 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb114103b90 msgr2=0x7fb114104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:24.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.025+0000 7fb1198b8640 1 --2- 172.21.15.46:0/1566518871 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb114103b90 0x7fb114104010 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb108009900 tx=0x7fb10802a410 comp rx=0 tx=0).stop 2024-04-26T21:32:24.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.025+0000 7fb1198b8640 1 -- 172.21.15.46:0/1566518871 shutdown_connections 2024-04-26T21:32:24.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.025+0000 7fb1198b8640 1 --2- 172.21.15.46:0/1566518871 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb114104550 0x7fb11410ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:24.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.025+0000 7fb1198b8640 1 --2- 172.21.15.46:0/1566518871 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb114103b90 0x7fb114104010 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:24.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.025+0000 7fb1198b8640 1 --2- 172.21.15.46:0/1566518871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1141029a0 0x7fb114102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:24.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.025+0000 7fb1198b8640 1 -- 172.21.15.46:0/1566518871 >> 172.21.15.46:0/1566518871 conn(0x7fb1140fdf30 msgr2=0x7fb114100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:24.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.025+0000 7fb1198b8640 1 -- 172.21.15.46:0/1566518871 shutdown_connections 2024-04-26T21:32:24.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.025+0000 7fb1198b8640 1 -- 172.21.15.46:0/1566518871 wait complete. 2024-04-26T21:32:24.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.025+0000 7fb1198b8640 1 Processor -- start 2024-04-26T21:32:24.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.026+0000 7fb1198b8640 1 -- start start 2024-04-26T21:32:24.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.026+0000 7fb1198b8640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1141029a0 0x7fb114076690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:24.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.026+0000 7fb1198b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb114103b90 0x7fb114074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:24.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.026+0000 7fb1198b8640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb114104550 0x7fb114075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:24.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.026+0000 7fb1198b8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb11410e4b0 con 0x7fb114103b90 2024-04-26T21:32:24.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.026+0000 7fb1198b8640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb11410e330 con 0x7fb1141029a0 2024-04-26T21:32:24.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.026+0000 7fb1198b8640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb11410e630 con 0x7fb114104550 2024-04-26T21:32:24.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.026+0000 7fb1127fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb114103b90 0x7fb114074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:24.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.026+0000 7fb1127fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb114103b90 0x7fb114074d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40062/0 (socket says 172.21.15.46:40062) 2024-04-26T21:32:24.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.026+0000 7fb1127fc640 1 -- 172.21.15.46:0/3582174924 learned_addr learned my addr 172.21.15.46:0/3582174924 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:24.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.026+0000 7fb1137fe640 1 --2- 172.21.15.46:0/3582174924 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb114104550 0x7fb114075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:24.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.026+0000 7fb112ffd640 1 --2- 172.21.15.46:0/3582174924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1141029a0 0x7fb114076690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:24.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.026+0000 7fb1127fc640 1 -- 172.21.15.46:0/3582174924 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb114104550 msgr2=0x7fb114075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:24.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.026+0000 7fb1127fc640 1 --2- 172.21.15.46:0/3582174924 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb114104550 0x7fb114075260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:24.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.026+0000 7fb1127fc640 1 -- 172.21.15.46:0/3582174924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1141029a0 msgr2=0x7fb114076690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:24.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.026+0000 7fb1127fc640 1 --2- 172.21.15.46:0/3582174924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1141029a0 0x7fb114076690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:24.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.026+0000 7fb1127fc640 1 -- 172.21.15.46:0/3582174924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb114075b20 con 0x7fb114103b90 2024-04-26T21:32:24.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.026+0000 7fb1137fe640 1 --2- 172.21.15.46:0/3582174924 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb114104550 0x7fb114075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:32:24.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.027+0000 7fb1127fc640 1 --2- 172.21.15.46:0/3582174924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb114103b90 0x7fb114074d20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb108005940 tx=0x7fb108004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:24.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.027+0000 7fb1188b6640 1 -- 172.21.15.46:0/3582174924 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10803f030 con 0x7fb114103b90 2024-04-26T21:32:24.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.027+0000 7fb1188b6640 1 -- 172.21.15.46:0/3582174924 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb108003cb0 con 0x7fb114103b90 2024-04-26T21:32:24.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.027+0000 7fb1198b8640 1 -- 172.21.15.46:0/3582174924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1141a2cd0 con 0x7fb114103b90 2024-04-26T21:32:24.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.027+0000 7fb1188b6640 1 -- 172.21.15.46:0/3582174924 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb108035070 con 0x7fb114103b90 2024-04-26T21:32:24.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.027+0000 7fb1198b8640 1 -- 172.21.15.46:0/3582174924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1141a3190 con 0x7fb114103b90 2024-04-26T21:32:24.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.028+0000 7fb1198b8640 1 -- 172.21.15.46:0/3582174924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb114109de0 con 0x7fb114103b90 2024-04-26T21:32:24.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.030+0000 7fb1188b6640 1 -- 172.21.15.46:0/3582174924 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb108042030 con 0x7fb114103b90 2024-04-26T21:32:24.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.031+0000 7fb1188b6640 1 --2- 172.21.15.46:0/3582174924 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb0fc0a3bb0 0x7fb0fc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:24.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.031+0000 7fb112ffd640 1 --2- 172.21.15.46:0/3582174924 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb0fc0a3bb0 0x7fb0fc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:24.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.031+0000 7fb1188b6640 1 -- 172.21.15.46:0/3582174924 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb1080f0990 con 0x7fb114103b90 2024-04-26T21:32:24.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.031+0000 7fb112ffd640 1 --2- 172.21.15.46:0/3582174924 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb0fc0a3bb0 0x7fb0fc0a6070 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb104005d00 tx=0x7fb104005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:24.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.036+0000 7fb1188b6640 1 -- 172.21.15.46:0/3582174924 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb108003e60 con 0x7fb114103b90 2024-04-26T21:32:24.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:23 smithi050 ceph-mon[32753]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:24.242 INFO:journalctl@ceph.mds.c.smithi142.stdout:Apr 26 21:32:23 smithi142 podman[63863]: 2024-04-26 21:32:23.890297349 +0000 UTC m=+0.018206710 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:32:24.242 INFO:journalctl@ceph.mds.c.smithi142.stdout:Apr 26 21:32:24 smithi142 podman[63863]: 2024-04-26 21:32:24.006403837 +0000 UTC m=+0.134313186 container create 365169d6c55a96dd3500b05a864caed7338878e6e50b1dc7695cbc75e707cce4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-c, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:32:24.242 INFO:journalctl@ceph.mds.c.smithi142.stdout:Apr 26 21:32:24 smithi142 podman[63863]: 2024-04-26 21:32:24.242555924 +0000 UTC m=+0.370465273 container init 365169d6c55a96dd3500b05a864caed7338878e6e50b1dc7695cbc75e707cce4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-c, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-26T21:32:24.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.263+0000 7fb1198b8640 1 -- 172.21.15.46:0/3582174924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb114063530 con 0x7fb114103b90 2024-04-26T21:32:24.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.263+0000 7fb1188b6640 1 -- 172.21.15.46:0/3582174924 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5518 (secure 0 0 0) 0x7fb1080bd770 con 0x7fb114103b90 2024-04-26T21:32:24.265 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:32:24.265 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3674296286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3674296286},{"type":"v1","addr":"172.21.15.46:6835","nonce":3674296286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/391791206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":391791206},{"type":"v1","addr":"172.21.15.46:6837","nonce":391791206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3621399933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3621399933},{"type":"v1","addr":"172.21.15.142:6833","nonce":3621399933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/345015778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":345015778},{"type":"v1","addr":"172.21.15.50:6833","nonce":345015778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/2546509863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":2546509863},{"type":"v1","addr":"172.21.15.50:6835","nonce":2546509863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/3791468066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":3791468066},{"type":"v1","addr":"172.21.15.142:6835","nonce":3791468066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-26T21:32:24.265 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 7 2024-04-26T21:32:24.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.266+0000 7fb1198b8640 1 -- 172.21.15.46:0/3582174924 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb0fc0a3bb0 msgr2=0x7fb0fc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:24.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.266+0000 7fb1198b8640 1 --2- 172.21.15.46:0/3582174924 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb0fc0a3bb0 0x7fb0fc0a6070 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb104005d00 tx=0x7fb104005c50 comp rx=0 tx=0).stop 2024-04-26T21:32:24.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.266+0000 7fb1198b8640 1 -- 172.21.15.46:0/3582174924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb114103b90 msgr2=0x7fb114074d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:24.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.266+0000 7fb1198b8640 1 --2- 172.21.15.46:0/3582174924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb114103b90 0x7fb114074d20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb108005940 tx=0x7fb108004430 comp rx=0 tx=0).stop 2024-04-26T21:32:24.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.266+0000 7fb1198b8640 1 -- 172.21.15.46:0/3582174924 shutdown_connections 2024-04-26T21:32:24.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.266+0000 7fb1198b8640 1 --2- 172.21.15.46:0/3582174924 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb0fc0a3bb0 0x7fb0fc0a6070 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:24.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.266+0000 7fb1198b8640 1 --2- 172.21.15.46:0/3582174924 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb114104550 0x7fb114075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:24.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.266+0000 7fb1198b8640 1 --2- 172.21.15.46:0/3582174924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb114103b90 0x7fb114074d20 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:24.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.266+0000 7fb1198b8640 1 --2- 172.21.15.46:0/3582174924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1141029a0 0x7fb114076690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:24.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.266+0000 7fb1198b8640 1 -- 172.21.15.46:0/3582174924 >> 172.21.15.46:0/3582174924 conn(0x7fb1140fdf30 msgr2=0x7fb114106fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:24.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.266+0000 7fb1198b8640 1 -- 172.21.15.46:0/3582174924 shutdown_connections 2024-04-26T21:32:24.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:24.266+0000 7fb1198b8640 1 -- 172.21.15.46:0/3582174924 wait complete. 2024-04-26T21:32:24.541 INFO:journalctl@ceph.mds.c.smithi142.stdout:Apr 26 21:32:24 smithi142 podman[63863]: 2024-04-26 21:32:24.245007815 +0000 UTC m=+0.372917173 container start 365169d6c55a96dd3500b05a864caed7338878e6e50b1dc7695cbc75e707cce4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-c, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-26T21:32:24.541 INFO:journalctl@ceph.mds.c.smithi142.stdout:Apr 26 21:32:24 smithi142 ceph-mds[63881]: -- 172.21.15.142:0/3897763293 <== mon.1 v2:172.21.15.142:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c429dcba40 con 0x55c429d9f000 2024-04-26T21:32:24.541 INFO:journalctl@ceph.mds.c.smithi142.stdout:Apr 26 21:32:24 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-c[63877]: starting mds.c at 2024-04-26T21:32:24.541 INFO:journalctl@ceph.mds.c.smithi142.stdout:Apr 26 21:32:24 smithi142 bash[63863]: 365169d6c55a96dd3500b05a864caed7338878e6e50b1dc7695cbc75e707cce4 2024-04-26T21:32:24.541 INFO:journalctl@ceph.mds.c.smithi142.stdout:Apr 26 21:32:24 smithi142 systemd[1]: Started Ceph mds.c for 36c7013a-0413-11ef-bc93-c7b262605968. 2024-04-26T21:32:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:24 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3582174924' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:24 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:24 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:24 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:24 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:24 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:25.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:24 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:25.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:24 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3582174924' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:25.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:24 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:25.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:24 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:25.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:24 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:25.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:24 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:25.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:24 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:25.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:24 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:25.102 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:24 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3582174924' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:25.103 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:24 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:25.103 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:24 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:25.103 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:24 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:25.103 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:24 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:25.103 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:24 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:25.103 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:24 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:26.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:25 smithi142 ceph-mon[32944]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:26.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:25 smithi142 ceph-mon[32944]: Deploying daemon mds.h on smithi050 2024-04-26T21:32:26.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:25 smithi142 ceph-mon[32944]: mds.? [v2:172.21.15.142:6836/1723132553,v1:172.21.15.142:6837/1723132553] up:boot 2024-04-26T21:32:26.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:25 smithi142 ceph-mon[32944]: fsmap 7 up:standby 2024-04-26T21:32:26.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:25 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T21:32:26.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:25 smithi046 ceph-mon[29201]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:26.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:25 smithi046 ceph-mon[29201]: Deploying daemon mds.h on smithi050 2024-04-26T21:32:26.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:25 smithi046 ceph-mon[29201]: mds.? [v2:172.21.15.142:6836/1723132553,v1:172.21.15.142:6837/1723132553] up:boot 2024-04-26T21:32:26.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:25 smithi046 ceph-mon[29201]: fsmap 7 up:standby 2024-04-26T21:32:26.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:25 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T21:32:26.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:25 smithi050 ceph-mon[32753]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:26.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:25 smithi050 ceph-mon[32753]: Deploying daemon mds.h on smithi050 2024-04-26T21:32:26.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:25 smithi050 ceph-mon[32753]: mds.? [v2:172.21.15.142:6836/1723132553,v1:172.21.15.142:6837/1723132553] up:boot 2024-04-26T21:32:26.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:25 smithi050 ceph-mon[32753]: fsmap 7 up:standby 2024-04-26T21:32:26.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:25 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T21:32:26.279 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:32:26.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.394+0000 7f9f09d1b640 1 -- 172.21.15.46:0/1522280934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f04103410 msgr2=0x7f9f04105800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:26.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.394+0000 7f9f09d1b640 1 --2- 172.21.15.46:0/1522280934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f04103410 0x7f9f04105800 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4009900 tx=0x7f9ef402a410 comp rx=0 tx=0).stop 2024-04-26T21:32:26.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.394+0000 7f9f09d1b640 1 -- 172.21.15.46:0/1522280934 shutdown_connections 2024-04-26T21:32:26.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.394+0000 7f9f09d1b640 1 --2- 172.21.15.46:0/1522280934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f04108670 0x7f9f0410aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:26.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.394+0000 7f9f09d1b640 1 --2- 172.21.15.46:0/1522280934 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f04105d40 0x7f9f04108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:26.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.394+0000 7f9f09d1b640 1 --2- 172.21.15.46:0/1522280934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f04103410 0x7f9f04105800 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:26.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.394+0000 7f9f09d1b640 1 -- 172.21.15.46:0/1522280934 >> 172.21.15.46:0/1522280934 conn(0x7f9f040fce20 msgr2=0x7f9f040ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:26.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.395+0000 7f9f09d1b640 1 -- 172.21.15.46:0/1522280934 shutdown_connections 2024-04-26T21:32:26.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.395+0000 7f9f09d1b640 1 -- 172.21.15.46:0/1522280934 wait complete. 2024-04-26T21:32:26.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.395+0000 7f9f09d1b640 1 Processor -- start 2024-04-26T21:32:26.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.395+0000 7f9f09d1b640 1 -- start start 2024-04-26T21:32:26.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.395+0000 7f9f09d1b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f04103410 0x7f9f04198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:26.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.395+0000 7f9f09d1b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f04105d40 0x7f9f04199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:26.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.395+0000 7f9f09d1b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f04108670 0x7f9f041a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:26.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.395+0000 7f9f09d1b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0410e530 con 0x7f9f04103410 2024-04-26T21:32:26.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.396+0000 7f9f09d1b640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0410e3b0 con 0x7f9f04105d40 2024-04-26T21:32:26.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.396+0000 7f9f09d1b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0410e6b0 con 0x7f9f04108670 2024-04-26T21:32:26.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.396+0000 7f9f037fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f04103410 0x7f9f04198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:26.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.396+0000 7f9f037fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f04103410 0x7f9f04198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52948/0 (socket says 172.21.15.46:52948) 2024-04-26T21:32:26.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.396+0000 7f9f037fe640 1 -- 172.21.15.46:0/3878308683 learned_addr learned my addr 172.21.15.46:0/3878308683 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:26.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.396+0000 7f9f02ffd640 1 --2- 172.21.15.46:0/3878308683 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f04105d40 0x7f9f04199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.396+0000 7f9f03fff640 1 --2- 172.21.15.46:0/3878308683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f04108670 0x7f9f041a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.396+0000 7f9f037fe640 1 -- 172.21.15.46:0/3878308683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f04108670 msgr2=0x7f9f041a01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.396+0000 7f9f037fe640 1 --2- 172.21.15.46:0/3878308683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f04108670 0x7f9f041a01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.396+0000 7f9f037fe640 1 -- 172.21.15.46:0/3878308683 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f04105d40 msgr2=0x7f9f04199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.396+0000 7f9f037fe640 1 --2- 172.21.15.46:0/3878308683 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f04105d40 0x7f9f04199190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.396+0000 7f9f037fe640 1 -- 172.21.15.46:0/3878308683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f041a0860 con 0x7f9f04103410 2024-04-26T21:32:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.396+0000 7f9f02ffd640 1 --2- 172.21.15.46:0/3878308683 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f04105d40 0x7f9f04199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:32:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.396+0000 7f9f037fe640 1 --2- 172.21.15.46:0/3878308683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f04103410 0x7f9f04198c50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4005ab0 tx=0x7f9ef4003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.396+0000 7f9f00ff9640 1 -- 172.21.15.46:0/3878308683 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef403f070 con 0x7f9f04103410 2024-04-26T21:32:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.396+0000 7f9f00ff9640 1 -- 172.21.15.46:0/3878308683 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ef403b460 con 0x7f9f04103410 2024-04-26T21:32:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.396+0000 7f9f09d1b640 1 -- 172.21.15.46:0/3878308683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f041a0af0 con 0x7f9f04103410 2024-04-26T21:32:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.396+0000 7f9f00ff9640 1 -- 172.21.15.46:0/3878308683 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef4035030 con 0x7f9f04103410 2024-04-26T21:32:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.396+0000 7f9f09d1b640 1 -- 172.21.15.46:0/3878308683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f041a0fe0 con 0x7f9f04103410 2024-04-26T21:32:26.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.397+0000 7f9f09d1b640 1 -- 172.21.15.46:0/3878308683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f040fe910 con 0x7f9f04103410 2024-04-26T21:32:26.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.398+0000 7f9f00ff9640 1 -- 172.21.15.46:0/3878308683 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9ef4042070 con 0x7f9f04103410 2024-04-26T21:32:26.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.399+0000 7f9f00ff9640 1 --2- 172.21.15.46:0/3878308683 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f9eec0a3bb0 0x7f9eec0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:26.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.399+0000 7f9f00ff9640 1 -- 172.21.15.46:0/3878308683 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9ef40f09a0 con 0x7f9f04103410 2024-04-26T21:32:26.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.399+0000 7f9f02ffd640 1 --2- 172.21.15.46:0/3878308683 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f9eec0a3bb0 0x7f9eec0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:26.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.400+0000 7f9f02ffd640 1 --2- 172.21.15.46:0/3878308683 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f9eec0a3bb0 0x7f9eec0a6070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9ef80058c0 tx=0x7f9ef8005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:26.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.404+0000 7f9f00ff9640 1 -- 172.21.15.46:0/3878308683 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9ef40bd820 con 0x7f9f04103410 2024-04-26T21:32:26.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.632+0000 7f9f09d1b640 1 -- 172.21.15.46:0/3878308683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f0402dd20 con 0x7f9f04103410 2024-04-26T21:32:26.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.633+0000 7f9f00ff9640 1 -- 172.21.15.46:0/3878308683 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6346 (secure 0 0 0) 0x7f9ef40bd120 con 0x7f9f04103410 2024-04-26T21:32:26.634 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:32:26.634 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3674296286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3674296286},{"type":"v1","addr":"172.21.15.46:6835","nonce":3674296286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/391791206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":391791206},{"type":"v1","addr":"172.21.15.46:6837","nonce":391791206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3621399933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3621399933},{"type":"v1","addr":"172.21.15.142:6833","nonce":3621399933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/345015778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":345015778},{"type":"v1","addr":"172.21.15.50:6833","nonce":345015778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/2546509863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":2546509863},{"type":"v1","addr":"172.21.15.50:6835","nonce":2546509863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/3791468066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":3791468066},{"type":"v1","addr":"172.21.15.142:6835","nonce":3791468066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/1723132553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1723132553},{"type":"v1","addr":"172.21.15.142:6837","nonce":1723132553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-26T21:32:26.635 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 8 2024-04-26T21:32:26.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.635+0000 7f9f09d1b640 1 -- 172.21.15.46:0/3878308683 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f9eec0a3bb0 msgr2=0x7f9eec0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:26.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.635+0000 7f9f09d1b640 1 --2- 172.21.15.46:0/3878308683 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f9eec0a3bb0 0x7f9eec0a6070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9ef80058c0 tx=0x7f9ef8005850 comp rx=0 tx=0).stop 2024-04-26T21:32:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.635+0000 7f9f09d1b640 1 -- 172.21.15.46:0/3878308683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f04103410 msgr2=0x7f9f04198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.635+0000 7f9f09d1b640 1 --2- 172.21.15.46:0/3878308683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f04103410 0x7f9f04198c50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4005ab0 tx=0x7f9ef4003cd0 comp rx=0 tx=0).stop 2024-04-26T21:32:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.635+0000 7f9f09d1b640 1 -- 172.21.15.46:0/3878308683 shutdown_connections 2024-04-26T21:32:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.635+0000 7f9f09d1b640 1 --2- 172.21.15.46:0/3878308683 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f9eec0a3bb0 0x7f9eec0a6070 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.635+0000 7f9f09d1b640 1 --2- 172.21.15.46:0/3878308683 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f04108670 0x7f9f041a01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.635+0000 7f9f09d1b640 1 --2- 172.21.15.46:0/3878308683 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f04105d40 0x7f9f04199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.635+0000 7f9f09d1b640 1 --2- 172.21.15.46:0/3878308683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f04103410 0x7f9f04198c50 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.635+0000 7f9f09d1b640 1 -- 172.21.15.46:0/3878308683 >> 172.21.15.46:0/3878308683 conn(0x7f9f040fce20 msgr2=0x7f9f04103e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.635+0000 7f9f09d1b640 1 -- 172.21.15.46:0/3878308683 shutdown_connections 2024-04-26T21:32:26.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:26.635+0000 7f9f09d1b640 1 -- 172.21.15.46:0/3878308683 wait complete. 2024-04-26T21:32:26.685 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:26 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3878308683' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:27.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:26 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3878308683' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:27.046 INFO:journalctl@ceph.mds.h.smithi050.stdout:Apr 26 21:32:26 smithi050 systemd[1]: Starting Ceph mds.h for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:32:27.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:26 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3878308683' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:27.315 INFO:journalctl@ceph.mds.h.smithi050.stdout:Apr 26 21:32:27 smithi050 podman[62724]: 2024-04-26 21:32:26.949239619 +0000 UTC m=+0.018680626 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:32:27.315 INFO:journalctl@ceph.mds.h.smithi050.stdout:Apr 26 21:32:27 smithi050 podman[62724]: 2024-04-26 21:32:27.068882405 +0000 UTC m=+0.138323399 container create 84e5daa6aa3301d06da29da5be8a926ab8a0fa4d8f229de78639a8ad32e32a19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS) 2024-04-26T21:32:27.315 INFO:journalctl@ceph.mds.h.smithi050.stdout:Apr 26 21:32:27 smithi050 podman[62724]: 2024-04-26 21:32:27.288275036 +0000 UTC m=+0.357716051 container init 84e5daa6aa3301d06da29da5be8a926ab8a0fa4d8f229de78639a8ad32e32a19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-h, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-26T21:32:27.315 INFO:journalctl@ceph.mds.h.smithi050.stdout:Apr 26 21:32:27 smithi050 podman[62724]: 2024-04-26 21:32:27.290885068 +0000 UTC m=+0.360326063 container start 84e5daa6aa3301d06da29da5be8a926ab8a0fa4d8f229de78639a8ad32e32a19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-h, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-26T21:32:27.315 INFO:journalctl@ceph.mds.h.smithi050.stdout:Apr 26 21:32:27 smithi050 ceph-mds[62743]: -- 172.21.15.50:0/2884586917 <== mon.2 v2:172.21.15.50:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eb08703a40 con 0x55eb086d6c00 2024-04-26T21:32:27.618 INFO:journalctl@ceph.mds.h.smithi050.stdout:Apr 26 21:32:27 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-h[62739]: starting mds.h at 2024-04-26T21:32:27.618 INFO:journalctl@ceph.mds.h.smithi050.stdout:Apr 26 21:32:27 smithi050 bash[62724]: 84e5daa6aa3301d06da29da5be8a926ab8a0fa4d8f229de78639a8ad32e32a19 2024-04-26T21:32:27.619 INFO:journalctl@ceph.mds.h.smithi050.stdout:Apr 26 21:32:27 smithi050 systemd[1]: Started Ceph mds.h for 36c7013a-0413-11ef-bc93-c7b262605968. 2024-04-26T21:32:28.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:27 smithi142 ceph-mon[32944]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:28.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:27 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:28.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:27 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:28.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:27 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:28.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:27 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:28.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:27 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:28.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:27 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:28.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:27 smithi046 ceph-mon[29201]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:28.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:27 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:28.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:27 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:28.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:27 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:28.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:27 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:28.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:27 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:28.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:27 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:28.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:27 smithi050 ceph-mon[32753]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:28.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:27 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:28.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:27 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:28.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:27 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:28.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:27 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:28.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:27 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:28.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:27 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:28.649 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:32:28.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.763+0000 7effd53e9640 1 -- 172.21.15.46:0/1984499131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effd0102a80 msgr2=0x7effd0102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:28.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.763+0000 7effd53e9640 1 --2- 172.21.15.46:0/1984499131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effd0102a80 0x7effd0102f00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7effb8009900 tx=0x7effb802a410 comp rx=0 tx=0).stop 2024-04-26T21:32:28.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.763+0000 7effd53e9640 1 -- 172.21.15.46:0/1984499131 shutdown_connections 2024-04-26T21:32:28.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.763+0000 7effd53e9640 1 --2- 172.21.15.46:0/1984499131 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effd0103440 0x7effd0109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:28.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.763+0000 7effd53e9640 1 --2- 172.21.15.46:0/1984499131 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effd0102a80 0x7effd0102f00 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:28.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.763+0000 7effd53e9640 1 --2- 172.21.15.46:0/1984499131 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7effd0101890 0x7effd0101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:28.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.763+0000 7effd53e9640 1 -- 172.21.15.46:0/1984499131 >> 172.21.15.46:0/1984499131 conn(0x7effd00fce20 msgr2=0x7effd00ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:28.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.763+0000 7effd53e9640 1 -- 172.21.15.46:0/1984499131 shutdown_connections 2024-04-26T21:32:28.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.763+0000 7effd53e9640 1 -- 172.21.15.46:0/1984499131 wait complete. 2024-04-26T21:32:28.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.764+0000 7effd53e9640 1 Processor -- start 2024-04-26T21:32:28.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.764+0000 7effd53e9640 1 -- start start 2024-04-26T21:32:28.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.764+0000 7effd53e9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effd0101890 0x7effd006bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:28.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.764+0000 7effd53e9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effd0102a80 0x7effd019d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:28.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.764+0000 7effd53e9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7effd0103440 0x7effd019ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:28.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.764+0000 7effd53e9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7effd010d410 con 0x7effd0101890 2024-04-26T21:32:28.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.764+0000 7effd53e9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7effd010d290 con 0x7effd0102a80 2024-04-26T21:32:28.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.764+0000 7effd53e9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7effd010d590 con 0x7effd0103440 2024-04-26T21:32:28.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.764+0000 7effceffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effd0101890 0x7effd006bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:28.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.764+0000 7effceffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effd0101890 0x7effd006bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52966/0 (socket says 172.21.15.46:52966) 2024-04-26T21:32:28.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.764+0000 7effceffd640 1 -- 172.21.15.46:0/4216283302 learned_addr learned my addr 172.21.15.46:0/4216283302 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:28.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.764+0000 7effc7fff640 1 --2- 172.21.15.46:0/4216283302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effd0102a80 0x7effd019d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:28.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.764+0000 7effcf7fe640 1 --2- 172.21.15.46:0/4216283302 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7effd0103440 0x7effd019ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:28.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.765+0000 7effceffd640 1 -- 172.21.15.46:0/4216283302 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7effd0103440 msgr2=0x7effd019ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:28.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.765+0000 7effceffd640 1 --2- 172.21.15.46:0/4216283302 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7effd0103440 0x7effd019ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:28.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.765+0000 7effceffd640 1 -- 172.21.15.46:0/4216283302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effd0102a80 msgr2=0x7effd019d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:28.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.765+0000 7effceffd640 1 --2- 172.21.15.46:0/4216283302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effd0102a80 0x7effd019d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:28.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.765+0000 7effceffd640 1 -- 172.21.15.46:0/4216283302 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effd01a0890 con 0x7effd0101890 2024-04-26T21:32:28.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.765+0000 7effcf7fe640 1 --2- 172.21.15.46:0/4216283302 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7effd0103440 0x7effd019ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:32:28.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.765+0000 7effc7fff640 1 --2- 172.21.15.46:0/4216283302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effd0102a80 0x7effd019d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:32:28.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.765+0000 7effceffd640 1 --2- 172.21.15.46:0/4216283302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effd0101890 0x7effd006bb00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7effc000d950 tx=0x7effc000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:28.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.765+0000 7effccff9640 1 -- 172.21.15.46:0/4216283302 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc0017070 con 0x7effd0101890 2024-04-26T21:32:28.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.765+0000 7effd53e9640 1 -- 172.21.15.46:0/4216283302 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effd01a0b20 con 0x7effd0101890 2024-04-26T21:32:28.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.765+0000 7effccff9640 1 -- 172.21.15.46:0/4216283302 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effc0009c30 con 0x7effd0101890 2024-04-26T21:32:28.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.765+0000 7effccff9640 1 -- 172.21.15.46:0/4216283302 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc000f620 con 0x7effd0101890 2024-04-26T21:32:28.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.765+0000 7effd53e9640 1 -- 172.21.15.46:0/4216283302 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effd01a0ff0 con 0x7effd0101890 2024-04-26T21:32:28.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.766+0000 7effd53e9640 1 -- 172.21.15.46:0/4216283302 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effd0108c80 con 0x7effd0101890 2024-04-26T21:32:28.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.767+0000 7effccff9640 1 -- 172.21.15.46:0/4216283302 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7effc001a070 con 0x7effd0101890 2024-04-26T21:32:28.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.768+0000 7effccff9640 1 --2- 172.21.15.46:0/4216283302 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7effbc0a3bb0 0x7effbc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:28.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.768+0000 7effc7fff640 1 --2- 172.21.15.46:0/4216283302 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7effbc0a3bb0 0x7effbc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:28.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.768+0000 7effccff9640 1 -- 172.21.15.46:0/4216283302 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7effc00cfaf0 con 0x7effd0101890 2024-04-26T21:32:28.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.768+0000 7effc7fff640 1 --2- 172.21.15.46:0/4216283302 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7effbc0a3bb0 0x7effbc0a6070 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7effb8000c00 tx=0x7effb80314c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:28.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:28.772+0000 7effccff9640 1 -- 172.21.15.46:0/4216283302 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7effc0009de0 con 0x7effd0101890 2024-04-26T21:32:29.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:29.000+0000 7effd53e9640 1 -- 172.21.15.46:0/4216283302 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7effd00635d0 con 0x7effd0101890 2024-04-26T21:32:29.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:29.001+0000 7effccff9640 1 -- 172.21.15.46:0/4216283302 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7171 (secure 0 0 0) 0x7effc009c8d0 con 0x7effd0101890 2024-04-26T21:32:29.002 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:32:29.003 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3674296286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3674296286},{"type":"v1","addr":"172.21.15.46:6835","nonce":3674296286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/391791206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":391791206},{"type":"v1","addr":"172.21.15.46:6837","nonce":391791206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3621399933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3621399933},{"type":"v1","addr":"172.21.15.142:6833","nonce":3621399933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/345015778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":345015778},{"type":"v1","addr":"172.21.15.50:6833","nonce":345015778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/2546509863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":2546509863},{"type":"v1","addr":"172.21.15.50:6835","nonce":2546509863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/3791468066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":3791468066},{"type":"v1","addr":"172.21.15.142:6835","nonce":3791468066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/1723132553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1723132553},{"type":"v1","addr":"172.21.15.142:6837","nonce":1723132553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24509,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/3189620395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3189620395},{"type":"v1","addr":"172.21.15.50:6837","nonce":3189620395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-26T21:32:29.003 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 9 2024-04-26T21:32:29.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:29.003+0000 7effd53e9640 1 -- 172.21.15.46:0/4216283302 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7effbc0a3bb0 msgr2=0x7effbc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:29.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:29.003+0000 7effd53e9640 1 --2- 172.21.15.46:0/4216283302 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7effbc0a3bb0 0x7effbc0a6070 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7effb8000c00 tx=0x7effb80314c0 comp rx=0 tx=0).stop 2024-04-26T21:32:29.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:29.003+0000 7effd53e9640 1 -- 172.21.15.46:0/4216283302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effd0101890 msgr2=0x7effd006bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:29.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:29.003+0000 7effd53e9640 1 --2- 172.21.15.46:0/4216283302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effd0101890 0x7effd006bb00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7effc000d950 tx=0x7effc000de10 comp rx=0 tx=0).stop 2024-04-26T21:32:29.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:29.003+0000 7effd53e9640 1 -- 172.21.15.46:0/4216283302 shutdown_connections 2024-04-26T21:32:29.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:29.003+0000 7effd53e9640 1 --2- 172.21.15.46:0/4216283302 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7effbc0a3bb0 0x7effbc0a6070 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:29.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:29.003+0000 7effd53e9640 1 --2- 172.21.15.46:0/4216283302 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7effd0103440 0x7effd019ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:29.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:29.003+0000 7effd53e9640 1 --2- 172.21.15.46:0/4216283302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7effd0102a80 0x7effd019d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:29.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:29.003+0000 7effd53e9640 1 --2- 172.21.15.46:0/4216283302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7effd0101890 0x7effd006bb00 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:29.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:29.003+0000 7effd53e9640 1 -- 172.21.15.46:0/4216283302 >> 172.21.15.46:0/4216283302 conn(0x7effd00fce20 msgr2=0x7effd0105e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:29.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:29.003+0000 7effd53e9640 1 -- 172.21.15.46:0/4216283302 shutdown_connections 2024-04-26T21:32:29.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:29.003+0000 7effd53e9640 1 -- 172.21.15.46:0/4216283302 wait complete. 2024-04-26T21:32:29.014 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:28 smithi046 ceph-mon[29201]: Deploying daemon mds.i on smithi142 2024-04-26T21:32:29.014 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:28 smithi046 ceph-mon[29201]: mds.? [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] up:boot 2024-04-26T21:32:29.014 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:28 smithi046 ceph-mon[29201]: fsmap 8 up:standby 2024-04-26T21:32:29.014 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:28 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T21:32:29.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:28 smithi142 ceph-mon[32944]: Deploying daemon mds.i on smithi142 2024-04-26T21:32:29.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:28 smithi142 ceph-mon[32944]: mds.? [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] up:boot 2024-04-26T21:32:29.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:28 smithi142 ceph-mon[32944]: fsmap 8 up:standby 2024-04-26T21:32:29.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:28 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T21:32:29.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:28 smithi050 ceph-mon[32753]: Deploying daemon mds.i on smithi142 2024-04-26T21:32:29.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:28 smithi050 ceph-mon[32753]: mds.? [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] up:boot 2024-04-26T21:32:29.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:28 smithi050 ceph-mon[32753]: fsmap 8 up:standby 2024-04-26T21:32:29.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:28 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T21:32:29.956 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:29 smithi142 ceph-mon[32944]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:29.956 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:29 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/4216283302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:30.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:29 smithi046 ceph-mon[29201]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:30.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:29 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/4216283302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:30.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:29 smithi050 ceph-mon[32753]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:30.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:29 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/4216283302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:30.540 INFO:journalctl@ceph.mds.i.smithi142.stdout:Apr 26 21:32:30 smithi142 systemd[1]: Started Ceph mds.i for 36c7013a-0413-11ef-bc93-c7b262605968. 2024-04-26T21:32:31.016 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:32:31.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.124+0000 7fa9ac2f9640 1 -- 172.21.15.46:0/1635060117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9a4103440 msgr2=0x7fa9a4109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:31.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.124+0000 7fa9ac2f9640 1 --2- 172.21.15.46:0/1635060117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9a4103440 0x7fa9a4109cd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa99c009ca0 tx=0x7fa99c02a8c0 comp rx=0 tx=0).stop 2024-04-26T21:32:31.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.124+0000 7fa9ac2f9640 1 -- 172.21.15.46:0/1635060117 shutdown_connections 2024-04-26T21:32:31.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.124+0000 7fa9ac2f9640 1 --2- 172.21.15.46:0/1635060117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9a4103440 0x7fa9a4109cd0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:31.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.124+0000 7fa9ac2f9640 1 --2- 172.21.15.46:0/1635060117 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9a4102a80 0x7fa9a4102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:31.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.124+0000 7fa9ac2f9640 1 --2- 172.21.15.46:0/1635060117 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9a4101890 0x7fa9a4101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:31.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.124+0000 7fa9ac2f9640 1 -- 172.21.15.46:0/1635060117 >> 172.21.15.46:0/1635060117 conn(0x7fa9a40fce20 msgr2=0x7fa9a40ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:31.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.124+0000 7fa9ac2f9640 1 -- 172.21.15.46:0/1635060117 shutdown_connections 2024-04-26T21:32:31.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.124+0000 7fa9ac2f9640 1 -- 172.21.15.46:0/1635060117 wait complete. 2024-04-26T21:32:31.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.125+0000 7fa9ac2f9640 1 Processor -- start 2024-04-26T21:32:31.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.125+0000 7fa9ac2f9640 1 -- start start 2024-04-26T21:32:31.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.125+0000 7fa9ac2f9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9a4101890 0x7fa9a406bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:31.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.125+0000 7fa9ac2f9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9a4102a80 0x7fa9a419d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:31.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.125+0000 7fa9ac2f9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9a4103440 0x7fa9a419feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:31.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.125+0000 7fa9ac2f9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a410d3a0 con 0x7fa9a4101890 2024-04-26T21:32:31.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.125+0000 7fa9ac2f9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a410d220 con 0x7fa9a4102a80 2024-04-26T21:32:31.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.125+0000 7fa9ac2f9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a410d520 con 0x7fa9a4103440 2024-04-26T21:32:31.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.125+0000 7fa9aa095640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9a4101890 0x7fa9a406bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:31.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.125+0000 7fa9aa095640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9a4101890 0x7fa9a406bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52982/0 (socket says 172.21.15.46:52982) 2024-04-26T21:32:31.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.125+0000 7fa9aa095640 1 -- 172.21.15.46:0/4252880479 learned_addr learned my addr 172.21.15.46:0/4252880479 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:31.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.125+0000 7fa9aa896640 1 --2- 172.21.15.46:0/4252880479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9a4103440 0x7fa9a419feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:31.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.125+0000 7fa9a9894640 1 --2- 172.21.15.46:0/4252880479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9a4102a80 0x7fa9a419d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:31.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.126+0000 7fa9aa095640 1 -- 172.21.15.46:0/4252880479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9a4103440 msgr2=0x7fa9a419feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:31.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.126+0000 7fa9aa095640 1 --2- 172.21.15.46:0/4252880479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9a4103440 0x7fa9a419feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:31.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.126+0000 7fa9aa095640 1 -- 172.21.15.46:0/4252880479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9a4102a80 msgr2=0x7fa9a419d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:31.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.126+0000 7fa9aa095640 1 --2- 172.21.15.46:0/4252880479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9a4102a80 0x7fa9a419d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:31.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.126+0000 7fa9aa095640 1 -- 172.21.15.46:0/4252880479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9a41a0770 con 0x7fa9a4101890 2024-04-26T21:32:31.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.126+0000 7fa9aa095640 1 --2- 172.21.15.46:0/4252880479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9a4101890 0x7fa9a406bab0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa9a000d950 tx=0x7fa9a000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:31.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.126+0000 7fa99b7fe640 1 -- 172.21.15.46:0/4252880479 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0017020 con 0x7fa9a4101890 2024-04-26T21:32:31.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.126+0000 7fa9ac2f9640 1 -- 172.21.15.46:0/4252880479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9a41a0a60 con 0x7fa9a4101890 2024-04-26T21:32:31.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.126+0000 7fa99b7fe640 1 -- 172.21.15.46:0/4252880479 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9a0009c30 con 0x7fa9a4101890 2024-04-26T21:32:31.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.126+0000 7fa99b7fe640 1 -- 172.21.15.46:0/4252880479 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a000f6b0 con 0x7fa9a4101890 2024-04-26T21:32:31.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.126+0000 7fa9ac2f9640 1 -- 172.21.15.46:0/4252880479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9a41a1000 con 0x7fa9a4101890 2024-04-26T21:32:31.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.127+0000 7fa9ac2f9640 1 -- 172.21.15.46:0/4252880479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9a4108c80 con 0x7fa9a4101890 2024-04-26T21:32:31.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.128+0000 7fa99b7fe640 1 -- 172.21.15.46:0/4252880479 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa9a001a020 con 0x7fa9a4101890 2024-04-26T21:32:31.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.128+0000 7fa99b7fe640 1 --2- 172.21.15.46:0/4252880479 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fa9900a3bb0 0x7fa9900a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:31.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.129+0000 7fa99b7fe640 1 -- 172.21.15.46:0/4252880479 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa9a00cfbf0 con 0x7fa9a4101890 2024-04-26T21:32:31.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.129+0000 7fa9a9894640 1 --2- 172.21.15.46:0/4252880479 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fa9900a3bb0 0x7fa9900a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:31.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.129+0000 7fa9a9894640 1 --2- 172.21.15.46:0/4252880479 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fa9900a3bb0 0x7fa9900a6070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa994005fd0 tx=0x7fa99400bf30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:31.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.133+0000 7fa99b7fe640 1 -- 172.21.15.46:0/4252880479 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa9a0009de0 con 0x7fa9a4101890 2024-04-26T21:32:31.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.366+0000 7fa9ac2f9640 1 -- 172.21.15.46:0/4252880479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9a402dca0 con 0x7fa9a4101890 2024-04-26T21:32:31.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.366+0000 7fa99b7fe640 1 -- 172.21.15.46:0/4252880479 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8001 (secure 0 0 0) 0x7fa9a009ca70 con 0x7fa9a4101890 2024-04-26T21:32:31.368 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:32:31.368 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3674296286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3674296286},{"type":"v1","addr":"172.21.15.46:6835","nonce":3674296286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/391791206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":391791206},{"type":"v1","addr":"172.21.15.46:6837","nonce":391791206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3621399933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3621399933},{"type":"v1","addr":"172.21.15.142:6833","nonce":3621399933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/345015778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":345015778},{"type":"v1","addr":"172.21.15.50:6833","nonce":345015778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/2546509863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":2546509863},{"type":"v1","addr":"172.21.15.50:6835","nonce":2546509863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/3791468066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":3791468066},{"type":"v1","addr":"172.21.15.142:6835","nonce":3791468066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/1723132553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1723132553},{"type":"v1","addr":"172.21.15.142:6837","nonce":1723132553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24509,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/3189620395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3189620395},{"type":"v1","addr":"172.21.15.50:6837","nonce":3189620395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24526,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6839/3089509101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3089509101},{"type":"v1","addr":"172.21.15.142:6839","nonce":3089509101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-26T21:32:31.368 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 10 2024-04-26T21:32:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.368+0000 7fa9ac2f9640 1 -- 172.21.15.46:0/4252880479 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fa9900a3bb0 msgr2=0x7fa9900a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.368+0000 7fa9ac2f9640 1 --2- 172.21.15.46:0/4252880479 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fa9900a3bb0 0x7fa9900a6070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa994005fd0 tx=0x7fa99400bf30 comp rx=0 tx=0).stop 2024-04-26T21:32:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.368+0000 7fa9ac2f9640 1 -- 172.21.15.46:0/4252880479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9a4101890 msgr2=0x7fa9a406bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.368+0000 7fa9ac2f9640 1 --2- 172.21.15.46:0/4252880479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9a4101890 0x7fa9a406bab0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa9a000d950 tx=0x7fa9a000de10 comp rx=0 tx=0).stop 2024-04-26T21:32:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.368+0000 7fa9ac2f9640 1 -- 172.21.15.46:0/4252880479 shutdown_connections 2024-04-26T21:32:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.368+0000 7fa9ac2f9640 1 --2- 172.21.15.46:0/4252880479 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fa9900a3bb0 0x7fa9900a6070 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.368+0000 7fa9ac2f9640 1 --2- 172.21.15.46:0/4252880479 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa9a4103440 0x7fa9a419feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.368+0000 7fa9ac2f9640 1 --2- 172.21.15.46:0/4252880479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9a4102a80 0x7fa9a419d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.368+0000 7fa9ac2f9640 1 --2- 172.21.15.46:0/4252880479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa9a4101890 0x7fa9a406bab0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.368+0000 7fa9ac2f9640 1 -- 172.21.15.46:0/4252880479 >> 172.21.15.46:0/4252880479 conn(0x7fa9a40fce20 msgr2=0x7fa9a4105e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.369+0000 7fa9ac2f9640 1 -- 172.21.15.46:0/4252880479 shutdown_connections 2024-04-26T21:32:31.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:31.369+0000 7fa9ac2f9640 1 -- 172.21.15.46:0/4252880479 wait complete. 2024-04-26T21:32:31.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:31 smithi142 ceph-mon[32944]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:31.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:31.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:31.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:31.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:31.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:31.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:31.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:31 smithi142 ceph-mon[32944]: Deploying daemon mds.g on smithi046 2024-04-26T21:32:31.542 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:31 smithi142 ceph-mon[32944]: mds.? [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] up:boot 2024-04-26T21:32:31.542 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:31 smithi142 ceph-mon[32944]: fsmap 9 up:standby 2024-04-26T21:32:31.542 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T21:32:31.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:31 smithi046 ceph-mon[29201]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:31.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:31.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:31.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:31.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:31.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:31.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:31.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:31 smithi046 ceph-mon[29201]: Deploying daemon mds.g on smithi046 2024-04-26T21:32:31.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:31 smithi046 ceph-mon[29201]: mds.? [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] up:boot 2024-04-26T21:32:31.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:31 smithi046 ceph-mon[29201]: fsmap 9 up:standby 2024-04-26T21:32:31.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T21:32:31.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:31 smithi050 ceph-mon[32753]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:31.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:31.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:31.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:31.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:31.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:31.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:31.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:31 smithi050 ceph-mon[32753]: Deploying daemon mds.g on smithi046 2024-04-26T21:32:31.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:31 smithi050 ceph-mon[32753]: mds.? [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] up:boot 2024-04-26T21:32:31.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:31 smithi050 ceph-mon[32753]: fsmap 9 up:standby 2024-04-26T21:32:31.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T21:32:32.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:32 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/4252880479' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:32.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:32 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/4252880479' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:32.554 INFO:journalctl@ceph.mds.g.smithi046.stdout:Apr 26 21:32:32 smithi046 systemd[1]: Starting Ceph mds.g for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:32:32.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:32 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/4252880479' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:32.954 INFO:journalctl@ceph.mds.g.smithi046.stdout:Apr 26 21:32:32 smithi046 podman[70229]: 2024-04-26 21:32:32.571063576 +0000 UTC m=+0.018506158 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:32:32.954 INFO:journalctl@ceph.mds.g.smithi046.stdout:Apr 26 21:32:32 smithi046 podman[70229]: 2024-04-26 21:32:32.692535854 +0000 UTC m=+0.139978469 container create d4aed68ced68dbf421a1d2678c48c5e269f5f9b4b8d862d0ad6d786e1d7bf4a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-g, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-26T21:32:32.954 INFO:journalctl@ceph.mds.g.smithi046.stdout:Apr 26 21:32:32 smithi046 podman[70229]: 2024-04-26 21:32:32.953981664 +0000 UTC m=+0.401424234 container init d4aed68ced68dbf421a1d2678c48c5e269f5f9b4b8d862d0ad6d786e1d7bf4a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-g, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-26T21:32:33.240 INFO:journalctl@ceph.mds.g.smithi046.stdout:Apr 26 21:32:32 smithi046 podman[70229]: 2024-04-26 21:32:32.95672792 +0000 UTC m=+0.404170502 container start d4aed68ced68dbf421a1d2678c48c5e269f5f9b4b8d862d0ad6d786e1d7bf4a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-g, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424) 2024-04-26T21:32:33.240 INFO:journalctl@ceph.mds.g.smithi046.stdout:Apr 26 21:32:32 smithi046 ceph-mds[70248]: -- 172.21.15.46:0/64792454 <== mon.0 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eac9721a40 con 0x55eac96f4c00 2024-04-26T21:32:33.240 INFO:journalctl@ceph.mds.g.smithi046.stdout:Apr 26 21:32:32 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-g[70244]: starting mds.g at 2024-04-26T21:32:33.240 INFO:journalctl@ceph.mds.g.smithi046.stdout:Apr 26 21:32:33 smithi046 bash[70229]: d4aed68ced68dbf421a1d2678c48c5e269f5f9b4b8d862d0ad6d786e1d7bf4a3 2024-04-26T21:32:33.240 INFO:journalctl@ceph.mds.g.smithi046.stdout:Apr 26 21:32:33 smithi046 systemd[1]: Started Ceph mds.g for 36c7013a-0413-11ef-bc93-c7b262605968. 2024-04-26T21:32:33.381 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:32:33.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.488+0000 7f77ef714640 1 -- 172.21.15.46:0/4243475960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77e8102a60 msgr2=0x7f77e8102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:33.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.488+0000 7f77ef714640 1 --2- 172.21.15.46:0/4243475960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77e8102a60 0x7f77e8102ee0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f77d8009900 tx=0x7f77d802a410 comp rx=0 tx=0).stop 2024-04-26T21:32:33.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.488+0000 7f77ef714640 1 -- 172.21.15.46:0/4243475960 shutdown_connections 2024-04-26T21:32:33.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.488+0000 7f77ef714640 1 --2- 172.21.15.46:0/4243475960 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77e8103420 0x7f77e8109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.488+0000 7f77ef714640 1 --2- 172.21.15.46:0/4243475960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77e8102a60 0x7f77e8102ee0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.488+0000 7f77ef714640 1 --2- 172.21.15.46:0/4243475960 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f77e8101870 0x7f77e8101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.488+0000 7f77ef714640 1 -- 172.21.15.46:0/4243475960 >> 172.21.15.46:0/4243475960 conn(0x7f77e80fcde0 msgr2=0x7f77e80ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:33.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.488+0000 7f77ef714640 1 -- 172.21.15.46:0/4243475960 shutdown_connections 2024-04-26T21:32:33.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.488+0000 7f77ef714640 1 -- 172.21.15.46:0/4243475960 wait complete. 2024-04-26T21:32:33.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.489+0000 7f77ef714640 1 Processor -- start 2024-04-26T21:32:33.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.489+0000 7f77ef714640 1 -- start start 2024-04-26T21:32:33.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.489+0000 7f77ef714640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f77e8101870 0x7f77e819afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:33.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.489+0000 7f77ef714640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77e8102a60 0x7f77e819b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:33.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.489+0000 7f77ef714640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77e8103420 0x7f77e81a2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:33.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.489+0000 7f77ef714640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f77e810d490 con 0x7f77e8103420 2024-04-26T21:32:33.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.489+0000 7f77ef714640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f77e810d310 con 0x7f77e8102a60 2024-04-26T21:32:33.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.489+0000 7f77ef714640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f77e810d610 con 0x7f77e8101870 2024-04-26T21:32:33.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.489+0000 7f77edcb1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77e8103420 0x7f77e81a2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:33.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.489+0000 7f77edcb1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77e8103420 0x7f77e81a2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53026/0 (socket says 172.21.15.46:53026) 2024-04-26T21:32:33.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.489+0000 7f77edcb1640 1 -- 172.21.15.46:0/4113320392 learned_addr learned my addr 172.21.15.46:0/4113320392 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:33.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.489+0000 7f77ed4b0640 1 --2- 172.21.15.46:0/4113320392 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f77e8101870 0x7f77e819afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:33.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.489+0000 7f77eccaf640 1 --2- 172.21.15.46:0/4113320392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77e8102a60 0x7f77e819b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:33.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.489+0000 7f77edcb1640 1 -- 172.21.15.46:0/4113320392 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f77e8101870 msgr2=0x7f77e819afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:33.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.489+0000 7f77edcb1640 1 --2- 172.21.15.46:0/4113320392 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f77e8101870 0x7f77e819afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.489+0000 7f77edcb1640 1 -- 172.21.15.46:0/4113320392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77e8102a60 msgr2=0x7f77e819b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:33.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.489+0000 7f77edcb1640 1 --2- 172.21.15.46:0/4113320392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77e8102a60 0x7f77e819b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.489+0000 7f77edcb1640 1 -- 172.21.15.46:0/4113320392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77e81a2c00 con 0x7f77e8103420 2024-04-26T21:32:33.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.489+0000 7f77ed4b0640 1 --2- 172.21.15.46:0/4113320392 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f77e8101870 0x7f77e819afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:32:33.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.490+0000 7f77edcb1640 1 --2- 172.21.15.46:0/4113320392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77e8103420 0x7f77e81a2590 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f77e000b970 tx=0x7f77e000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:33.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.490+0000 7f77de7fc640 1 -- 172.21.15.46:0/4113320392 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e0019070 con 0x7f77e8103420 2024-04-26T21:32:33.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.490+0000 7f77de7fc640 1 -- 172.21.15.46:0/4113320392 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77e0014440 con 0x7f77e8103420 2024-04-26T21:32:33.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.490+0000 7f77ef714640 1 -- 172.21.15.46:0/4113320392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77e81a2ef0 con 0x7f77e8103420 2024-04-26T21:32:33.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.490+0000 7f77de7fc640 1 -- 172.21.15.46:0/4113320392 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e00075d0 con 0x7f77e8103420 2024-04-26T21:32:33.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.490+0000 7f77ef714640 1 -- 172.21.15.46:0/4113320392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77e81a3430 con 0x7f77e8103420 2024-04-26T21:32:33.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.491+0000 7f77ef714640 1 -- 172.21.15.46:0/4113320392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77e81a36c0 con 0x7f77e8103420 2024-04-26T21:32:33.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.492+0000 7f77de7fc640 1 -- 172.21.15.46:0/4113320392 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f77e001c070 con 0x7f77e8103420 2024-04-26T21:32:33.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.493+0000 7f77de7fc640 1 --2- 172.21.15.46:0/4113320392 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f77d40a3c30 0x7f77d40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:33.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.493+0000 7f77ed4b0640 1 --2- 172.21.15.46:0/4113320392 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f77d40a3c30 0x7f77d40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:33.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.493+0000 7f77de7fc640 1 -- 172.21.15.46:0/4113320392 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f77e00cf550 con 0x7f77e8103420 2024-04-26T21:32:33.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.493+0000 7f77ed4b0640 1 --2- 172.21.15.46:0/4113320392 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f77d40a3c30 0x7f77d40a60f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f77e81028c0 tx=0x7f77e4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:33.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.498+0000 7f77de7fc640 1 -- 172.21.15.46:0/4113320392 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f77e009c3d0 con 0x7f77e8103420 2024-04-26T21:32:33.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:33 smithi142 ceph-mon[32944]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:33.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:33 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:33.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:33 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:33.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:33 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:33.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:33 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:33.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:33 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:33.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:33 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:33.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:33 smithi046 ceph-mon[29201]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:33.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:33 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:33.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:33 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:33.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:33 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:33.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:33 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:33.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:33 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:33.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:33 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:33.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:33 smithi050 ceph-mon[32753]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:33.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:33 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:33.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:33 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:33.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:33 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:33.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:33 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:33.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:33 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:33.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:33 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:33.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.738+0000 7f77ef714640 1 -- 172.21.15.46:0/4113320392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f77e81a3890 con 0x7f77e8103420 2024-04-26T21:32:33.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.739+0000 7f77de7fc640 1 -- 172.21.15.46:0/4113320392 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8827 (secure 0 0 0) 0x7f77e009bcd0 con 0x7f77e8103420 2024-04-26T21:32:33.740 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:32:33.740 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3674296286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3674296286},{"type":"v1","addr":"172.21.15.46:6835","nonce":3674296286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/391791206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":391791206},{"type":"v1","addr":"172.21.15.46:6837","nonce":391791206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/1222489912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1222489912},{"type":"v1","addr":"172.21.15.46:6839","nonce":1222489912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3621399933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3621399933},{"type":"v1","addr":"172.21.15.142:6833","nonce":3621399933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/345015778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":345015778},{"type":"v1","addr":"172.21.15.50:6833","nonce":345015778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/2546509863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":2546509863},{"type":"v1","addr":"172.21.15.50:6835","nonce":2546509863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/3791468066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":3791468066},{"type":"v1","addr":"172.21.15.142:6835","nonce":3791468066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/1723132553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1723132553},{"type":"v1","addr":"172.21.15.142:6837","nonce":1723132553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24509,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/3189620395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3189620395},{"type":"v1","addr":"172.21.15.50:6837","nonce":3189620395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24526,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6839/3089509101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3089509101},{"type":"v1","addr":"172.21.15.142:6839","nonce":3089509101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-26T21:32:33.740 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 11 2024-04-26T21:32:33.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.741+0000 7f77ef714640 1 -- 172.21.15.46:0/4113320392 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f77d40a3c30 msgr2=0x7f77d40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:33.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.741+0000 7f77ef714640 1 --2- 172.21.15.46:0/4113320392 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f77d40a3c30 0x7f77d40a60f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f77e81028c0 tx=0x7f77e4008040 comp rx=0 tx=0).stop 2024-04-26T21:32:33.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.741+0000 7f77ef714640 1 -- 172.21.15.46:0/4113320392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77e8103420 msgr2=0x7f77e81a2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:33.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.741+0000 7f77ef714640 1 --2- 172.21.15.46:0/4113320392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77e8103420 0x7f77e81a2590 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f77e000b970 tx=0x7f77e000be30 comp rx=0 tx=0).stop 2024-04-26T21:32:33.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.741+0000 7f77ef714640 1 -- 172.21.15.46:0/4113320392 shutdown_connections 2024-04-26T21:32:33.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.741+0000 7f77ef714640 1 --2- 172.21.15.46:0/4113320392 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f77d40a3c30 0x7f77d40a60f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.741+0000 7f77ef714640 1 --2- 172.21.15.46:0/4113320392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77e8103420 0x7f77e81a2590 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.741+0000 7f77ef714640 1 --2- 172.21.15.46:0/4113320392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77e8102a60 0x7f77e819b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.741+0000 7f77ef714640 1 --2- 172.21.15.46:0/4113320392 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f77e8101870 0x7f77e819afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.741+0000 7f77ef714640 1 -- 172.21.15.46:0/4113320392 >> 172.21.15.46:0/4113320392 conn(0x7f77e80fcde0 msgr2=0x7f77e8069b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:33.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.741+0000 7f77ef714640 1 -- 172.21.15.46:0/4113320392 shutdown_connections 2024-04-26T21:32:33.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:33.741+0000 7f77ef714640 1 -- 172.21.15.46:0/4113320392 wait complete. 2024-04-26T21:32:34.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:34 smithi142 ceph-mon[32944]: Deploying daemon mds.a on smithi046 2024-04-26T21:32:34.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:34 smithi142 ceph-mon[32944]: mds.? [v2:172.21.15.46:6838/1222489912,v1:172.21.15.46:6839/1222489912] up:boot 2024-04-26T21:32:34.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:34 smithi142 ceph-mon[32944]: fsmap 10 up:standby 2024-04-26T21:32:34.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:34 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T21:32:34.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:34 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/4113320392' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:34.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:34 smithi046 ceph-mon[29201]: Deploying daemon mds.a on smithi046 2024-04-26T21:32:34.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:34 smithi046 ceph-mon[29201]: mds.? [v2:172.21.15.46:6838/1222489912,v1:172.21.15.46:6839/1222489912] up:boot 2024-04-26T21:32:34.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:34 smithi046 ceph-mon[29201]: fsmap 10 up:standby 2024-04-26T21:32:34.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:34 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T21:32:34.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:34 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/4113320392' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:34.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:34 smithi050 ceph-mon[32753]: Deploying daemon mds.a on smithi046 2024-04-26T21:32:34.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:34 smithi050 ceph-mon[32753]: mds.? [v2:172.21.15.46:6838/1222489912,v1:172.21.15.46:6839/1222489912] up:boot 2024-04-26T21:32:34.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:34 smithi050 ceph-mon[32753]: fsmap 10 up:standby 2024-04-26T21:32:34.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:34 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T21:32:34.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:34 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/4113320392' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:35.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:35 smithi142 ceph-mon[32944]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:35.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:35 smithi046 ceph-mon[29201]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:35.554 INFO:journalctl@ceph.mds.a.smithi046.stdout:Apr 26 21:32:35 smithi046 systemd[1]: Starting Ceph mds.a for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:32:35.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:35 smithi050 ceph-mon[32753]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:35.753 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:32:35.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.862+0000 7fb2f36c8640 1 -- 172.21.15.46:0/1660336463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2ec1086b0 msgr2=0x7fb2ec10aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:35.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.862+0000 7fb2f36c8640 1 --2- 172.21.15.46:0/1660336463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2ec1086b0 0x7fb2ec10aaa0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4009ca0 tx=0x7fb2e402a8c0 comp rx=0 tx=0).stop 2024-04-26T21:32:35.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.862+0000 7fb2f36c8640 1 -- 172.21.15.46:0/1660336463 shutdown_connections 2024-04-26T21:32:35.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.862+0000 7fb2f36c8640 1 --2- 172.21.15.46:0/1660336463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2ec1086b0 0x7fb2ec10aaa0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:35.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.862+0000 7fb2f36c8640 1 --2- 172.21.15.46:0/1660336463 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2ec105d80 0x7fb2ec108170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:35.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.862+0000 7fb2f36c8640 1 --2- 172.21.15.46:0/1660336463 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2ec103450 0x7fb2ec105840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:35.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.862+0000 7fb2f36c8640 1 -- 172.21.15.46:0/1660336463 >> 172.21.15.46:0/1660336463 conn(0x7fb2ec0fcde0 msgr2=0x7fb2ec0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:35.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.862+0000 7fb2f36c8640 1 -- 172.21.15.46:0/1660336463 shutdown_connections 2024-04-26T21:32:35.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.862+0000 7fb2f36c8640 1 -- 172.21.15.46:0/1660336463 wait complete. 2024-04-26T21:32:35.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.863+0000 7fb2f36c8640 1 Processor -- start 2024-04-26T21:32:35.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.863+0000 7fb2f36c8640 1 -- start start 2024-04-26T21:32:35.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.863+0000 7fb2f36c8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2ec103450 0x7fb2ec102dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:35.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.863+0000 7fb2f36c8640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2ec105d80 0x7fb2ec101400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:35.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.863+0000 7fb2f36c8640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2ec1086b0 0x7fb2ec101940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:35.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.863+0000 7fb2f36c8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ec10e510 con 0x7fb2ec103450 2024-04-26T21:32:35.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.863+0000 7fb2f36c8640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ec10e390 con 0x7fb2ec1086b0 2024-04-26T21:32:35.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.863+0000 7fb2f36c8640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ec10e690 con 0x7fb2ec105d80 2024-04-26T21:32:35.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.863+0000 7fb2f1464640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2ec103450 0x7fb2ec102dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:35.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.863+0000 7fb2f1464640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2ec103450 0x7fb2ec102dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42026/0 (socket says 172.21.15.46:42026) 2024-04-26T21:32:35.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.863+0000 7fb2f1464640 1 -- 172.21.15.46:0/1167468602 learned_addr learned my addr 172.21.15.46:0/1167468602 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:35.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.863+0000 7fb2f0c63640 1 --2- 172.21.15.46:0/1167468602 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2ec105d80 0x7fb2ec101400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:35.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.863+0000 7fb2f1c65640 1 --2- 172.21.15.46:0/1167468602 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2ec1086b0 0x7fb2ec101940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:35.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.864+0000 7fb2f1464640 1 -- 172.21.15.46:0/1167468602 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2ec105d80 msgr2=0x7fb2ec101400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:35.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.864+0000 7fb2f1464640 1 --2- 172.21.15.46:0/1167468602 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2ec105d80 0x7fb2ec101400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:35.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.864+0000 7fb2f1464640 1 -- 172.21.15.46:0/1167468602 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2ec1086b0 msgr2=0x7fb2ec101940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:35.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.864+0000 7fb2f1464640 1 --2- 172.21.15.46:0/1167468602 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2ec1086b0 0x7fb2ec101940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:35.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.864+0000 7fb2f1464640 1 -- 172.21.15.46:0/1167468602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2ec1021d0 con 0x7fb2ec103450 2024-04-26T21:32:35.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.864+0000 7fb2f1c65640 1 --2- 172.21.15.46:0/1167468602 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2ec1086b0 0x7fb2ec101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:32:35.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.864+0000 7fb2f1464640 1 --2- 172.21.15.46:0/1167468602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2ec103450 0x7fb2ec102dc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb2e800c920 tx=0x7fb2e800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:35.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.864+0000 7fb2e27fc640 1 -- 172.21.15.46:0/1167468602 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8007c70 con 0x7fb2ec103450 2024-04-26T21:32:35.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.864+0000 7fb2e27fc640 1 -- 172.21.15.46:0/1167468602 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2e8007e10 con 0x7fb2ec103450 2024-04-26T21:32:35.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.864+0000 7fb2f36c8640 1 -- 172.21.15.46:0/1167468602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2ec1a2cb0 con 0x7fb2ec103450 2024-04-26T21:32:35.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.864+0000 7fb2e27fc640 1 -- 172.21.15.46:0/1167468602 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8018820 con 0x7fb2ec103450 2024-04-26T21:32:35.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.864+0000 7fb2f36c8640 1 -- 172.21.15.46:0/1167468602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2ec1a31b0 con 0x7fb2ec103450 2024-04-26T21:32:35.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.865+0000 7fb2f36c8640 1 -- 172.21.15.46:0/1167468602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2ec0fe8d0 con 0x7fb2ec103450 2024-04-26T21:32:35.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.866+0000 7fb2e27fc640 1 -- 172.21.15.46:0/1167468602 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb2e800f450 con 0x7fb2ec103450 2024-04-26T21:32:35.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.867+0000 7fb2e27fc640 1 --2- 172.21.15.46:0/1167468602 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb2d80a3bb0 0x7fb2d80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:35.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.867+0000 7fb2f0c63640 1 --2- 172.21.15.46:0/1167468602 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb2d80a3bb0 0x7fb2d80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:35.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.867+0000 7fb2e27fc640 1 -- 172.21.15.46:0/1167468602 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb2e80d0420 con 0x7fb2ec103450 2024-04-26T21:32:35.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.867+0000 7fb2f0c63640 1 --2- 172.21.15.46:0/1167468602 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb2d80a3bb0 0x7fb2d80a6070 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec102b30 tx=0x7fb2dc008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:35.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:35.870+0000 7fb2e27fc640 1 -- 172.21.15.46:0/1167468602 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb2e8014030 con 0x7fb2ec103450 2024-04-26T21:32:35.936 INFO:journalctl@ceph.mds.a.smithi046.stdout:Apr 26 21:32:35 smithi046 podman[70576]: 2024-04-26 21:32:35.556919034 +0000 UTC m=+0.018814197 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:32:35.937 INFO:journalctl@ceph.mds.a.smithi046.stdout:Apr 26 21:32:35 smithi046 podman[70576]: 2024-04-26 21:32:35.691150325 +0000 UTC m=+0.153045462 container create 28ef6aefc492b0a2aebe8d10781d281f7ec3123620077d55ded6c72f7a42505c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-26T21:32:35.937 INFO:journalctl@ceph.mds.a.smithi046.stdout:Apr 26 21:32:35 smithi046 podman[70576]: 2024-04-26 21:32:35.936297785 +0000 UTC m=+0.398192943 container init 28ef6aefc492b0a2aebe8d10781d281f7ec3123620077d55ded6c72f7a42505c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T21:32:36.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:36.103+0000 7fb2f36c8640 1 -- 172.21.15.46:0/1167468602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2ec02dad0 con 0x7fb2ec103450 2024-04-26T21:32:36.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:36.104+0000 7fb2e27fc640 1 -- 172.21.15.46:0/1167468602 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8827 (secure 0 0 0) 0x7fb2e8016300 con 0x7fb2ec103450 2024-04-26T21:32:36.105 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:32:36.106 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3674296286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3674296286},{"type":"v1","addr":"172.21.15.46:6835","nonce":3674296286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/391791206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":391791206},{"type":"v1","addr":"172.21.15.46:6837","nonce":391791206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/1222489912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1222489912},{"type":"v1","addr":"172.21.15.46:6839","nonce":1222489912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3621399933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3621399933},{"type":"v1","addr":"172.21.15.142:6833","nonce":3621399933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/345015778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":345015778},{"type":"v1","addr":"172.21.15.50:6833","nonce":345015778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/2546509863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":2546509863},{"type":"v1","addr":"172.21.15.50:6835","nonce":2546509863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/3791468066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":3791468066},{"type":"v1","addr":"172.21.15.142:6835","nonce":3791468066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/1723132553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1723132553},{"type":"v1","addr":"172.21.15.142:6837","nonce":1723132553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24509,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/3189620395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3189620395},{"type":"v1","addr":"172.21.15.50:6837","nonce":3189620395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24526,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6839/3089509101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3089509101},{"type":"v1","addr":"172.21.15.142:6839","nonce":3089509101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-26T21:32:36.106 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 11 2024-04-26T21:32:36.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:36.106+0000 7fb2f36c8640 1 -- 172.21.15.46:0/1167468602 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb2d80a3bb0 msgr2=0x7fb2d80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:36.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:36.106+0000 7fb2f36c8640 1 --2- 172.21.15.46:0/1167468602 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb2d80a3bb0 0x7fb2d80a6070 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec102b30 tx=0x7fb2dc008040 comp rx=0 tx=0).stop 2024-04-26T21:32:36.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:36.106+0000 7fb2f36c8640 1 -- 172.21.15.46:0/1167468602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2ec103450 msgr2=0x7fb2ec102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:36.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:36.106+0000 7fb2f36c8640 1 --2- 172.21.15.46:0/1167468602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2ec103450 0x7fb2ec102dc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb2e800c920 tx=0x7fb2e800cde0 comp rx=0 tx=0).stop 2024-04-26T21:32:36.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:36.106+0000 7fb2f36c8640 1 -- 172.21.15.46:0/1167468602 shutdown_connections 2024-04-26T21:32:36.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:36.106+0000 7fb2f36c8640 1 --2- 172.21.15.46:0/1167468602 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb2d80a3bb0 0x7fb2d80a6070 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:36.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:36.106+0000 7fb2f36c8640 1 --2- 172.21.15.46:0/1167468602 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2ec1086b0 0x7fb2ec101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:36.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:36.106+0000 7fb2f36c8640 1 --2- 172.21.15.46:0/1167468602 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2ec105d80 0x7fb2ec101400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:36.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:36.106+0000 7fb2f36c8640 1 --2- 172.21.15.46:0/1167468602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2ec103450 0x7fb2ec102dc0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:36.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:36.106+0000 7fb2f36c8640 1 -- 172.21.15.46:0/1167468602 >> 172.21.15.46:0/1167468602 conn(0x7fb2ec0fcde0 msgr2=0x7fb2ec103d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:36.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:36.107+0000 7fb2f36c8640 1 -- 172.21.15.46:0/1167468602 shutdown_connections 2024-04-26T21:32:36.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:36.107+0000 7fb2f36c8640 1 -- 172.21.15.46:0/1167468602 wait complete. 2024-04-26T21:32:36.253 INFO:journalctl@ceph.mds.a.smithi046.stdout:Apr 26 21:32:35 smithi046 podman[70576]: 2024-04-26 21:32:35.939248805 +0000 UTC m=+0.401143941 container start 28ef6aefc492b0a2aebe8d10781d281f7ec3123620077d55ded6c72f7a42505c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-a, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:32:36.253 INFO:journalctl@ceph.mds.a.smithi046.stdout:Apr 26 21:32:35 smithi046 ceph-mds[70634]: -- 172.21.15.46:0/17238429 <== mon.0 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560b24e7ba40 con 0x560b24e4e800 2024-04-26T21:32:36.253 INFO:journalctl@ceph.mds.a.smithi046.stdout:Apr 26 21:32:35 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-a[70611]: starting mds.a at 2024-04-26T21:32:36.253 INFO:journalctl@ceph.mds.a.smithi046.stdout:Apr 26 21:32:36 smithi046 bash[70576]: 28ef6aefc492b0a2aebe8d10781d281f7ec3123620077d55ded6c72f7a42505c 2024-04-26T21:32:36.253 INFO:journalctl@ceph.mds.a.smithi046.stdout:Apr 26 21:32:36 smithi046 systemd[1]: Started Ceph mds.a for 36c7013a-0413-11ef-bc93-c7b262605968. 2024-04-26T21:32:36.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:36 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/1167468602' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:36.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:36 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:36.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:36 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:36.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:36 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:36.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:36 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:36.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:36 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:36.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:36 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:36.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:36 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1167468602' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:36.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:36 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:36.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:36 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:36.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:36 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:36.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:36 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:36.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:36 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:36.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:36 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:36.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:36 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/1167468602' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:36.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:36 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:36.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:36 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:36.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:36 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:36.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:36 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:32:36.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:36 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:32:36.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:36 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:37.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:37 smithi142 ceph-mon[32944]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:37.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:37 smithi142 ceph-mon[32944]: Deploying daemon mds.b on smithi050 2024-04-26T21:32:37.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:37 smithi142 ceph-mon[32944]: mds.? [v2:172.21.15.46:6840/860605548,v1:172.21.15.46:6841/860605548] up:boot 2024-04-26T21:32:37.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:37 smithi142 ceph-mon[32944]: fsmap 11 up:standby 2024-04-26T21:32:37.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:37 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T21:32:37.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:37 smithi046 ceph-mon[29201]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:37.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:37 smithi046 ceph-mon[29201]: Deploying daemon mds.b on smithi050 2024-04-26T21:32:37.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:37 smithi046 ceph-mon[29201]: mds.? [v2:172.21.15.46:6840/860605548,v1:172.21.15.46:6841/860605548] up:boot 2024-04-26T21:32:37.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:37 smithi046 ceph-mon[29201]: fsmap 11 up:standby 2024-04-26T21:32:37.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:37 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T21:32:37.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:37 smithi050 ceph-mon[32753]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:37.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:37 smithi050 ceph-mon[32753]: Deploying daemon mds.b on smithi050 2024-04-26T21:32:37.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:37 smithi050 ceph-mon[32753]: mds.? [v2:172.21.15.46:6840/860605548,v1:172.21.15.46:6841/860605548] up:boot 2024-04-26T21:32:37.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:37 smithi050 ceph-mon[32753]: fsmap 11 up:standby 2024-04-26T21:32:37.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:37 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T21:32:38.120 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:32:38.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.234+0000 7f4366a6b640 1 -- 172.21.15.46:0/733005792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4360105d40 msgr2=0x7f4360108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:38.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.234+0000 7f4366a6b640 1 --2- 172.21.15.46:0/733005792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4360105d40 0x7f4360108130 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4350009900 tx=0x7f435002a3f0 comp rx=0 tx=0).stop 2024-04-26T21:32:38.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.234+0000 7f4366a6b640 1 -- 172.21.15.46:0/733005792 shutdown_connections 2024-04-26T21:32:38.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.234+0000 7f4366a6b640 1 --2- 172.21.15.46:0/733005792 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4360108670 0x7f436010aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:38.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.234+0000 7f4366a6b640 1 --2- 172.21.15.46:0/733005792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4360105d40 0x7f4360108130 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:38.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.234+0000 7f4366a6b640 1 --2- 172.21.15.46:0/733005792 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4360103410 0x7f4360105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:38.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.234+0000 7f4366a6b640 1 -- 172.21.15.46:0/733005792 >> 172.21.15.46:0/733005792 conn(0x7f43600fce20 msgr2=0x7f43600ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:38.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.234+0000 7f4366a6b640 1 -- 172.21.15.46:0/733005792 shutdown_connections 2024-04-26T21:32:38.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.234+0000 7f4366a6b640 1 -- 172.21.15.46:0/733005792 wait complete. 2024-04-26T21:32:38.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.235+0000 7f4366a6b640 1 Processor -- start 2024-04-26T21:32:38.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.235+0000 7f4366a6b640 1 -- start start 2024-04-26T21:32:38.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.235+0000 7f4366a6b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4360103410 0x7f4360198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:38.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.235+0000 7f4366a6b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4360105d40 0x7f4360199040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:38.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.235+0000 7f4366a6b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4360108670 0x7f43601a00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:38.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.235+0000 7f4366a6b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f436010e4b0 con 0x7f4360103410 2024-04-26T21:32:38.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.235+0000 7f4366a6b640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f436010e330 con 0x7f4360108670 2024-04-26T21:32:38.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.235+0000 7f4366a6b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f436010e630 con 0x7f4360105d40 2024-04-26T21:32:38.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.235+0000 7f4364807640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4360103410 0x7f4360198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:38.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.235+0000 7f4364807640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4360103410 0x7f4360198b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42052/0 (socket says 172.21.15.46:42052) 2024-04-26T21:32:38.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.235+0000 7f4364807640 1 -- 172.21.15.46:0/3185334059 learned_addr learned my addr 172.21.15.46:0/3185334059 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:38.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.235+0000 7f4365008640 1 --2- 172.21.15.46:0/3185334059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4360108670 0x7f43601a00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:38.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.235+0000 7f435ffff640 1 --2- 172.21.15.46:0/3185334059 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4360105d40 0x7f4360199040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:38.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.235+0000 7f4364807640 1 -- 172.21.15.46:0/3185334059 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4360105d40 msgr2=0x7f4360199040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:38.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.235+0000 7f4364807640 1 --2- 172.21.15.46:0/3185334059 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4360105d40 0x7f4360199040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:38.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.235+0000 7f4364807640 1 -- 172.21.15.46:0/3185334059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4360108670 msgr2=0x7f43601a00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:38.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.235+0000 7f4364807640 1 --2- 172.21.15.46:0/3185334059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4360108670 0x7f43601a00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:38.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.235+0000 7f4364807640 1 -- 172.21.15.46:0/3185334059 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43601a0720 con 0x7f4360103410 2024-04-26T21:32:38.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.235+0000 7f4365008640 1 --2- 172.21.15.46:0/3185334059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4360108670 0x7f43601a00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:32:38.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.236+0000 7f4364807640 1 --2- 172.21.15.46:0/3185334059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4360103410 0x7f4360198b00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f435800cce0 tx=0x7f435800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:38.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.236+0000 7f435dffb640 1 -- 172.21.15.46:0/3185334059 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435800f650 con 0x7f4360103410 2024-04-26T21:32:38.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.236+0000 7f435dffb640 1 -- 172.21.15.46:0/3185334059 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f435800fca0 con 0x7f4360103410 2024-04-26T21:32:38.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.236+0000 7f4366a6b640 1 -- 172.21.15.46:0/3185334059 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43601a0a10 con 0x7f4360103410 2024-04-26T21:32:38.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.236+0000 7f435dffb640 1 -- 172.21.15.46:0/3185334059 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4358018b40 con 0x7f4360103410 2024-04-26T21:32:38.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.236+0000 7f4366a6b640 1 -- 172.21.15.46:0/3185334059 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43601a0f60 con 0x7f4360103410 2024-04-26T21:32:38.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.237+0000 7f4366a6b640 1 -- 172.21.15.46:0/3185334059 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43600fe910 con 0x7f4360103410 2024-04-26T21:32:38.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.238+0000 7f435dffb640 1 -- 172.21.15.46:0/3185334059 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f435800f7f0 con 0x7f4360103410 2024-04-26T21:32:38.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.239+0000 7f435dffb640 1 --2- 172.21.15.46:0/3185334059 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f434c0a3bb0 0x7f434c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:38.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.239+0000 7f435ffff640 1 --2- 172.21.15.46:0/3185334059 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f434c0a3bb0 0x7f434c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:38.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.239+0000 7f435dffb640 1 -- 172.21.15.46:0/3185334059 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f43580d0580 con 0x7f4360103410 2024-04-26T21:32:38.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.240+0000 7f435ffff640 1 --2- 172.21.15.46:0/3185334059 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f434c0a3bb0 0x7f434c0a6070 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4350031500 tx=0x7f4350000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:38.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.244+0000 7f435dffb640 1 -- 172.21.15.46:0/3185334059 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4358014030 con 0x7f4360103410 2024-04-26T21:32:38.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.468+0000 7f4366a6b640 1 -- 172.21.15.46:0/3185334059 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4360063530 con 0x7f4360103410 2024-04-26T21:32:38.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.468+0000 7f435dffb640 1 -- 172.21.15.46:0/3185334059 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9650 (secure 0 0 0) 0x7f43580bbe20 con 0x7f4360103410 2024-04-26T21:32:38.469 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:32:38.470 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3674296286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3674296286},{"type":"v1","addr":"172.21.15.46:6835","nonce":3674296286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/391791206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":391791206},{"type":"v1","addr":"172.21.15.46:6837","nonce":391791206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/1222489912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1222489912},{"type":"v1","addr":"172.21.15.46:6839","nonce":1222489912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14658,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/860605548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":860605548},{"type":"v1","addr":"172.21.15.46:6841","nonce":860605548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3621399933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3621399933},{"type":"v1","addr":"172.21.15.142:6833","nonce":3621399933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/345015778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":345015778},{"type":"v1","addr":"172.21.15.50:6833","nonce":345015778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/2546509863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":2546509863},{"type":"v1","addr":"172.21.15.50:6835","nonce":2546509863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/3791468066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":3791468066},{"type":"v1","addr":"172.21.15.142:6835","nonce":3791468066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/1723132553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1723132553},{"type":"v1","addr":"172.21.15.142:6837","nonce":1723132553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24509,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/3189620395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3189620395},{"type":"v1","addr":"172.21.15.50:6837","nonce":3189620395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24526,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6839/3089509101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3089509101},{"type":"v1","addr":"172.21.15.142:6839","nonce":3089509101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-26T21:32:38.470 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 12 2024-04-26T21:32:38.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.471+0000 7f4366a6b640 1 -- 172.21.15.46:0/3185334059 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f434c0a3bb0 msgr2=0x7f434c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:38.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.471+0000 7f4366a6b640 1 --2- 172.21.15.46:0/3185334059 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f434c0a3bb0 0x7f434c0a6070 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4350031500 tx=0x7f4350000bc0 comp rx=0 tx=0).stop 2024-04-26T21:32:38.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.471+0000 7f4366a6b640 1 -- 172.21.15.46:0/3185334059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4360103410 msgr2=0x7f4360198b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:38.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.471+0000 7f4366a6b640 1 --2- 172.21.15.46:0/3185334059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4360103410 0x7f4360198b00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f435800cce0 tx=0x7f435800a620 comp rx=0 tx=0).stop 2024-04-26T21:32:38.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.471+0000 7f4366a6b640 1 -- 172.21.15.46:0/3185334059 shutdown_connections 2024-04-26T21:32:38.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.471+0000 7f4366a6b640 1 --2- 172.21.15.46:0/3185334059 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f434c0a3bb0 0x7f434c0a6070 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:38.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.471+0000 7f4366a6b640 1 --2- 172.21.15.46:0/3185334059 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4360108670 0x7f43601a00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:38.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.471+0000 7f4366a6b640 1 --2- 172.21.15.46:0/3185334059 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4360105d40 0x7f4360199040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:38.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.471+0000 7f4366a6b640 1 --2- 172.21.15.46:0/3185334059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4360103410 0x7f4360198b00 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:38.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.471+0000 7f4366a6b640 1 -- 172.21.15.46:0/3185334059 >> 172.21.15.46:0/3185334059 conn(0x7f43600fce20 msgr2=0x7f4360103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:38.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.471+0000 7f4366a6b640 1 -- 172.21.15.46:0/3185334059 shutdown_connections 2024-04-26T21:32:38.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:38.471+0000 7f4366a6b640 1 -- 172.21.15.46:0/3185334059 wait complete. 2024-04-26T21:32:38.559 INFO:journalctl@ceph.mds.b.smithi050.stdout:Apr 26 21:32:38 smithi050 systemd[1]: Starting Ceph mds.b for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:32:38.868 INFO:journalctl@ceph.mds.b.smithi050.stdout:Apr 26 21:32:38 smithi050 podman[63034]: 2024-04-26 21:32:38.681767817 +0000 UTC m=+0.018427326 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:32:38.868 INFO:journalctl@ceph.mds.b.smithi050.stdout:Apr 26 21:32:38 smithi050 podman[63034]: 2024-04-26 21:32:38.808090444 +0000 UTC m=+0.144749996 container create f0907047d44e3ba7d721dd63ac9a0079adc3d03d0fbff2f02e8c74e3afaf5ca1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-b, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-26T21:32:39.257 INFO:journalctl@ceph.mds.b.smithi050.stdout:Apr 26 21:32:39 smithi050 podman[63034]: 2024-04-26 21:32:39.044514671 +0000 UTC m=+0.381174178 container init f0907047d44e3ba7d721dd63ac9a0079adc3d03d0fbff2f02e8c74e3afaf5ca1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T21:32:39.257 INFO:journalctl@ceph.mds.b.smithi050.stdout:Apr 26 21:32:39 smithi050 podman[63034]: 2024-04-26 21:32:39.047208048 +0000 UTC m=+0.383867560 container start f0907047d44e3ba7d721dd63ac9a0079adc3d03d0fbff2f02e8c74e3afaf5ca1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-26T21:32:39.257 INFO:journalctl@ceph.mds.b.smithi050.stdout:Apr 26 21:32:39 smithi050 ceph-mds[63053]: -- 172.21.15.50:0/2205561121 <== mon.2 v2:172.21.15.50:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5636f3d45a40 con 0x5636f3d19000 2024-04-26T21:32:39.257 INFO:journalctl@ceph.mds.b.smithi050.stdout:Apr 26 21:32:39 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-b[63049]: starting mds.b at 2024-04-26T21:32:39.257 INFO:journalctl@ceph.mds.b.smithi050.stdout:Apr 26 21:32:39 smithi050 bash[63034]: f0907047d44e3ba7d721dd63ac9a0079adc3d03d0fbff2f02e8c74e3afaf5ca1 2024-04-26T21:32:39.257 INFO:journalctl@ceph.mds.b.smithi050.stdout:Apr 26 21:32:39 smithi050 systemd[1]: Started Ceph mds.b for 36c7013a-0413-11ef-bc93-c7b262605968. 2024-04-26T21:32:39.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:39 smithi142 ceph-mon[32944]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:39.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:39 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3185334059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:39.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:39 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:39.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:39 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:39.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:39 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:39.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:39 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:39.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:39 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:39.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:39 smithi046 ceph-mon[29201]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:39.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:39 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3185334059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:39.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:39 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:39.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:39 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:39.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:39 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:39.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:39 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:39.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:39 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:39.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:39 smithi050 ceph-mon[32753]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:39.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:39 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3185334059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:39.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:39 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:39.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:39 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:39.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:39 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:39.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:39 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:39.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:39 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:40.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:40 smithi142 ceph-mon[32944]: mds.? [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] up:boot 2024-04-26T21:32:40.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:40 smithi142 ceph-mon[32944]: fsmap 12 up:standby 2024-04-26T21:32:40.460 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:40 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T21:32:40.461 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:40 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:32:40.461 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:40 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:40.484 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:32:40.509 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:40 smithi046 ceph-mon[29201]: mds.? [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] up:boot 2024-04-26T21:32:40.509 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:40 smithi046 ceph-mon[29201]: fsmap 12 up:standby 2024-04-26T21:32:40.509 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:40 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T21:32:40.509 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:40 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:32:40.509 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:40 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:40.538 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:40 smithi050 ceph-mon[32753]: mds.? [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] up:boot 2024-04-26T21:32:40.538 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:40 smithi050 ceph-mon[32753]: fsmap 12 up:standby 2024-04-26T21:32:40.538 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:40 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T21:32:40.539 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:40 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:32:40.539 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:40 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:40.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.588+0000 7f2955340640 1 -- 172.21.15.46:0/2601059694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2950104550 msgr2=0x7f295010ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:40.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.588+0000 7f2955340640 1 --2- 172.21.15.46:0/2601059694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2950104550 0x7f295010ade0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2948009ca0 tx=0x7f294802a8c0 comp rx=0 tx=0).stop 2024-04-26T21:32:40.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.588+0000 7f2955340640 1 -- 172.21.15.46:0/2601059694 shutdown_connections 2024-04-26T21:32:40.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.588+0000 7f2955340640 1 --2- 172.21.15.46:0/2601059694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2950104550 0x7f295010ade0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:40.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.588+0000 7f2955340640 1 --2- 172.21.15.46:0/2601059694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2950103b90 0x7f2950104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:40.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.588+0000 7f2955340640 1 --2- 172.21.15.46:0/2601059694 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f29501029a0 0x7f2950102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:40.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.588+0000 7f2955340640 1 -- 172.21.15.46:0/2601059694 >> 172.21.15.46:0/2601059694 conn(0x7f29500fdf30 msgr2=0x7f2950100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:40.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.588+0000 7f2955340640 1 -- 172.21.15.46:0/2601059694 shutdown_connections 2024-04-26T21:32:40.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.588+0000 7f2955340640 1 -- 172.21.15.46:0/2601059694 wait complete. 2024-04-26T21:32:40.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.589+0000 7f2955340640 1 Processor -- start 2024-04-26T21:32:40.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.589+0000 7f2955340640 1 -- start start 2024-04-26T21:32:40.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.589+0000 7f2955340640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29501029a0 0x7f2950198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:40.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.589+0000 7f2955340640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2950103b90 0x7f2950199020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:40.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.589+0000 7f2955340640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2950104550 0x7f29501a0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:40.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.589+0000 7f2955340640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f295010e4d0 con 0x7f29501029a0 2024-04-26T21:32:40.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.589+0000 7f2955340640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f295010e350 con 0x7f2950103b90 2024-04-26T21:32:40.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.589+0000 7f2955340640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f295010e650 con 0x7f2950104550 2024-04-26T21:32:40.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.589+0000 7f294effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29501029a0 0x7f2950198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:40.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.589+0000 7f294effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29501029a0 0x7f2950198ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42074/0 (socket says 172.21.15.46:42074) 2024-04-26T21:32:40.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.589+0000 7f294effd640 1 -- 172.21.15.46:0/1971863550 learned_addr learned my addr 172.21.15.46:0/1971863550 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:40.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.589+0000 7f294e7fc640 1 --2- 172.21.15.46:0/1971863550 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2950103b90 0x7f2950199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:40.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.589+0000 7f294f7fe640 1 --2- 172.21.15.46:0/1971863550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2950104550 0x7f29501a0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:40.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.589+0000 7f294effd640 1 -- 172.21.15.46:0/1971863550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2950104550 msgr2=0x7f29501a0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:40.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.589+0000 7f294effd640 1 --2- 172.21.15.46:0/1971863550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2950104550 0x7f29501a0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:40.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.589+0000 7f294effd640 1 -- 172.21.15.46:0/1971863550 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2950103b90 msgr2=0x7f2950199020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:40.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.589+0000 7f294effd640 1 --2- 172.21.15.46:0/1971863550 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2950103b90 0x7f2950199020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:40.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.589+0000 7f294effd640 1 -- 172.21.15.46:0/1971863550 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29501a0700 con 0x7f29501029a0 2024-04-26T21:32:40.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.589+0000 7f294e7fc640 1 --2- 172.21.15.46:0/1971863550 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2950103b90 0x7f2950199020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:32:40.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.589+0000 7f294effd640 1 --2- 172.21.15.46:0/1971863550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29501029a0 0x7f2950198ae0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f294000e9c0 tx=0x7f294000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:40.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.590+0000 7f2937fff640 1 -- 172.21.15.46:0/1971863550 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294000cd80 con 0x7f29501029a0 2024-04-26T21:32:40.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.590+0000 7f2937fff640 1 -- 172.21.15.46:0/1971863550 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29400035f0 con 0x7f29501029a0 2024-04-26T21:32:40.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.590+0000 7f2955340640 1 -- 172.21.15.46:0/1971863550 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29501a09f0 con 0x7f29501029a0 2024-04-26T21:32:40.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.590+0000 7f2937fff640 1 -- 172.21.15.46:0/1971863550 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2940010690 con 0x7f29501029a0 2024-04-26T21:32:40.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.590+0000 7f2955340640 1 -- 172.21.15.46:0/1971863550 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29501a0f40 con 0x7f29501029a0 2024-04-26T21:32:40.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.591+0000 7f2955340640 1 -- 172.21.15.46:0/1971863550 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2950109cc0 con 0x7f29501029a0 2024-04-26T21:32:40.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.592+0000 7f2937fff640 1 -- 172.21.15.46:0/1971863550 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2940004000 con 0x7f29501029a0 2024-04-26T21:32:40.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.592+0000 7f2937fff640 1 --2- 172.21.15.46:0/1971863550 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f29380a3c30 0x7f29380a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:40.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.593+0000 7f294e7fc640 1 --2- 172.21.15.46:0/1971863550 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f29380a3c30 0x7f29380a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:40.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.593+0000 7f2937fff640 1 -- 172.21.15.46:0/1971863550 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f29400d0510 con 0x7f29501029a0 2024-04-26T21:32:40.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.593+0000 7f294e7fc640 1 --2- 172.21.15.46:0/1971863550 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f29380a3c30 0x7f29380a60f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2950199ff0 tx=0x7f294400a5c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:40.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.597+0000 7f2937fff640 1 -- 172.21.15.46:0/1971863550 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f294009d390 con 0x7f29501029a0 2024-04-26T21:32:40.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.846+0000 7f2955340640 1 -- 172.21.15.46:0/1971863550 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f295002dad0 con 0x7f29501029a0 2024-04-26T21:32:40.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.846+0000 7f2937fff640 1 -- 172.21.15.46:0/1971863550 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10476 (secure 0 0 0) 0x7f294009cc90 con 0x7f29501029a0 2024-04-26T21:32:40.848 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:32:40.848 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3674296286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3674296286},{"type":"v1","addr":"172.21.15.46:6835","nonce":3674296286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/391791206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":391791206},{"type":"v1","addr":"172.21.15.46:6837","nonce":391791206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/1222489912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1222489912},{"type":"v1","addr":"172.21.15.46:6839","nonce":1222489912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14658,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/860605548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":860605548},{"type":"v1","addr":"172.21.15.46:6841","nonce":860605548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3621399933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3621399933},{"type":"v1","addr":"172.21.15.142:6833","nonce":3621399933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/345015778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":345015778},{"type":"v1","addr":"172.21.15.50:6833","nonce":345015778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6835/2546509863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":2546509863},{"type":"v1","addr":"172.21.15.50:6835","nonce":2546509863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24493,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6835/3791468066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":3791468066},{"type":"v1","addr":"172.21.15.142:6835","nonce":3791468066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24505,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6837/1723132553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1723132553},{"type":"v1","addr":"172.21.15.142:6837","nonce":1723132553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24509,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6837/3189620395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3189620395},{"type":"v1","addr":"172.21.15.50:6837","nonce":3189620395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24526,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6839/3089509101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3089509101},{"type":"v1","addr":"172.21.15.142:6839","nonce":3089509101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6839/2453253368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2453253368},{"type":"v1","addr":"172.21.15.50:6839","nonce":2453253368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-26T21:32:40.848 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 13 2024-04-26T21:32:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.848+0000 7f2955340640 1 -- 172.21.15.46:0/1971863550 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f29380a3c30 msgr2=0x7f29380a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.848+0000 7f2955340640 1 --2- 172.21.15.46:0/1971863550 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f29380a3c30 0x7f29380a60f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2950199ff0 tx=0x7f294400a5c0 comp rx=0 tx=0).stop 2024-04-26T21:32:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.848+0000 7f2955340640 1 -- 172.21.15.46:0/1971863550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29501029a0 msgr2=0x7f2950198ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.848+0000 7f2955340640 1 --2- 172.21.15.46:0/1971863550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29501029a0 0x7f2950198ae0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f294000e9c0 tx=0x7f294000ee80 comp rx=0 tx=0).stop 2024-04-26T21:32:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.848+0000 7f2955340640 1 -- 172.21.15.46:0/1971863550 shutdown_connections 2024-04-26T21:32:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.848+0000 7f2955340640 1 --2- 172.21.15.46:0/1971863550 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f29380a3c30 0x7f29380a60f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.848+0000 7f2955340640 1 --2- 172.21.15.46:0/1971863550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2950104550 0x7f29501a0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.848+0000 7f2955340640 1 --2- 172.21.15.46:0/1971863550 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2950103b90 0x7f2950199020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.848+0000 7f2955340640 1 --2- 172.21.15.46:0/1971863550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29501029a0 0x7f2950198ae0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.848+0000 7f2955340640 1 -- 172.21.15.46:0/1971863550 >> 172.21.15.46:0/1971863550 conn(0x7f29500fdf30 msgr2=0x7f2950106e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:40.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.849+0000 7f2955340640 1 -- 172.21.15.46:0/1971863550 shutdown_connections 2024-04-26T21:32:40.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:40.849+0000 7f2955340640 1 -- 172.21.15.46:0/1971863550 wait complete. 2024-04-26T21:32:40.862 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:32:41.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.013+0000 7f677aaf1640 1 -- 172.21.15.46:0/4221704285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6774108650 msgr2=0x7f677410aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:41.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.013+0000 7f677aaf1640 1 --2- 172.21.15.46:0/4221704285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6774108650 0x7f677410aaa0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f676c009ca0 tx=0x7f676c02a8c0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.014+0000 7f677aaf1640 1 -- 172.21.15.46:0/4221704285 shutdown_connections 2024-04-26T21:32:41.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.014+0000 7f677aaf1640 1 --2- 172.21.15.46:0/4221704285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6774108650 0x7f677410aaa0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.014+0000 7f677aaf1640 1 --2- 172.21.15.46:0/4221704285 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6774105d20 0x7f6774108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.014+0000 7f677aaf1640 1 --2- 172.21.15.46:0/4221704285 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67741033f0 0x7f67741057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.014+0000 7f677aaf1640 1 -- 172.21.15.46:0/4221704285 >> 172.21.15.46:0/4221704285 conn(0x7f67740fcde0 msgr2=0x7f67740ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:41.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.014+0000 7f677aaf1640 1 -- 172.21.15.46:0/4221704285 shutdown_connections 2024-04-26T21:32:41.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.014+0000 7f677aaf1640 1 -- 172.21.15.46:0/4221704285 wait complete. 2024-04-26T21:32:41.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.014+0000 7f677aaf1640 1 Processor -- start 2024-04-26T21:32:41.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.014+0000 7f677aaf1640 1 -- start start 2024-04-26T21:32:41.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.014+0000 7f677aaf1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67741033f0 0x7f6774198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:41.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.014+0000 7f677aaf1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6774105d20 0x7f6774199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:41.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.014+0000 7f677aaf1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6774108650 0x7f67741a0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:41.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.014+0000 7f677aaf1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f677410e510 con 0x7f6774105d20 2024-04-26T21:32:41.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.014+0000 7f677aaf1640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f677410e390 con 0x7f67741033f0 2024-04-26T21:32:41.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.014+0000 7f677aaf1640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f677410e690 con 0x7f6774108650 2024-04-26T21:32:41.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.014+0000 7f6773fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6774105d20 0x7f6774199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:41.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.015+0000 7f677888d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67741033f0 0x7f6774198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:41.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.015+0000 7f677908e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6774108650 0x7f67741a0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:41.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.015+0000 7f6773fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6774105d20 0x7f6774199090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42078/0 (socket says 172.21.15.46:42078) 2024-04-26T21:32:41.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.015+0000 7f6773fff640 1 -- 172.21.15.46:0/2984770840 learned_addr learned my addr 172.21.15.46:0/2984770840 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:41.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.015+0000 7f677888d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67741033f0 0x7f6774198b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.46:53436/0 (socket says 172.21.15.46:53436) 2024-04-26T21:32:41.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.015+0000 7f677908e640 1 -- 172.21.15.46:0/2984770840 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67741033f0 msgr2=0x7f6774198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:41.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.015+0000 7f677908e640 1 --2- 172.21.15.46:0/2984770840 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67741033f0 0x7f6774198b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.015+0000 7f677908e640 1 -- 172.21.15.46:0/2984770840 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6774105d20 msgr2=0x7f6774199090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:41.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.015+0000 7f677908e640 1 --2- 172.21.15.46:0/2984770840 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6774105d20 0x7f6774199090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.015+0000 7f677908e640 1 -- 172.21.15.46:0/2984770840 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67741a0770 con 0x7f6774108650 2024-04-26T21:32:41.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.015+0000 7f677908e640 1 --2- 172.21.15.46:0/2984770840 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6774108650 0x7f67741a0100 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f676c031a20 tx=0x7f676c0316f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:41.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.015+0000 7f6771ffb640 1 -- 172.21.15.46:0/2984770840 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676c003b70 con 0x7f6774108650 2024-04-26T21:32:41.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.015+0000 7f6771ffb640 1 -- 172.21.15.46:0/2984770840 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f676c003d10 con 0x7f6774108650 2024-04-26T21:32:41.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.015+0000 7f677aaf1640 1 -- 172.21.15.46:0/2984770840 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67741a0a00 con 0x7f6774108650 2024-04-26T21:32:41.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.015+0000 7f6771ffb640 1 -- 172.21.15.46:0/2984770840 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676c02fae0 con 0x7f6774108650 2024-04-26T21:32:41.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.016+0000 7f677aaf1640 1 -- 172.21.15.46:0/2984770840 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67741a0f50 con 0x7f6774108650 2024-04-26T21:32:41.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.017+0000 7f677aaf1640 1 -- 172.21.15.46:0/2984770840 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67740fe8d0 con 0x7f6774108650 2024-04-26T21:32:41.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.018+0000 7f6771ffb640 1 -- 172.21.15.46:0/2984770840 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f676c004610 con 0x7f6774108650 2024-04-26T21:32:41.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.018+0000 7f6771ffb640 1 --2- 172.21.15.46:0/2984770840 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f67640a3bb0 0x7f67640a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:41.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.018+0000 7f6771ffb640 1 -- 172.21.15.46:0/2984770840 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f676c0f14e0 con 0x7f6774108650 2024-04-26T21:32:41.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.018+0000 7f677888d640 1 --2- 172.21.15.46:0/2984770840 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f67640a3bb0 0x7f67640a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:41.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.019+0000 7f677888d640 1 --2- 172.21.15.46:0/2984770840 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f67640a3bb0 0x7f67640a6070 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6768009bd0 tx=0x7f6768005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:41.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.024+0000 7f6771ffb640 1 -- 172.21.15.46:0/2984770840 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f676c0be2c0 con 0x7f6774108650 2024-04-26T21:32:41.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.222+0000 7f677aaf1640 1 -- 172.21.15.46:0/2984770840 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6774063530 con 0x7f6774108650 2024-04-26T21:32:41.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.224+0000 7f6771ffb640 1 -- 172.21.15.46:0/2984770840 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18434 (secure 0 0 0) 0x7f676c0bdbc0 con 0x7f6774108650 2024-04-26T21:32:41.225 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:32:41.225 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":81,"fsid":"36c7013a-0413-11ef-bc93-c7b262605968","created":"2024-04-26T21:24:30.481156+0000","modified":"2024-04-26T21:31:51.663297+0000","last_up_change":"2024-04-26T21:31:50.100491+0000","last_in_change":"2024-04-26T21:31:29.643428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:27:57.935988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f53c8702-e4da-435f-a9dc-3911d0d36ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6803","nonce":2762665125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6805","nonce":2762665125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6809","nonce":2762665125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6807","nonce":2762665125}]},"public_addr":"172.21.15.46:6803/2762665125","cluster_addr":"172.21.15.46:6805/2762665125","heartbeat_back_addr":"172.21.15.46:6809/2762665125","heartbeat_front_addr":"172.21.15.46:6807/2762665125","state":["exists","up"]},{"osd":1,"uuid":"fc4d6f52-fc8a-4efd-80c9-c092b57e7f42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6801","nonce":1628008970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6803","nonce":1628008970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6807","nonce":1628008970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6805","nonce":1628008970}]},"public_addr":"172.21.15.50:6801/1628008970","cluster_addr":"172.21.15.50:6803/1628008970","heartbeat_back_addr":"172.21.15.50:6807/1628008970","heartbeat_front_addr":"172.21.15.50:6805/1628008970","state":["exists","up"]},{"osd":2,"uuid":"26d1baa1-4ea1-4b7e-9fcf-4f8bbf7329c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6801","nonce":1513279137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6803","nonce":1513279137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6807","nonce":1513279137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6805","nonce":1513279137}]},"public_addr":"172.21.15.142:6801/1513279137","cluster_addr":"172.21.15.142:6803/1513279137","heartbeat_back_addr":"172.21.15.142:6807/1513279137","heartbeat_front_addr":"172.21.15.142:6805/1513279137","state":["exists","up"]},{"osd":3,"uuid":"34d2401f-5fcc-491c-8fcf-c3c492611f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6811","nonce":2435181760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6813","nonce":2435181760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6817","nonce":2435181760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6815","nonce":2435181760}]},"public_addr":"172.21.15.46:6811/2435181760","cluster_addr":"172.21.15.46:6813/2435181760","heartbeat_back_addr":"172.21.15.46:6817/2435181760","heartbeat_front_addr":"172.21.15.46:6815/2435181760","state":["exists","up"]},{"osd":4,"uuid":"986bb709-e4cf-400d-982e-c3f5ac413e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6809","nonce":3177333250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6811","nonce":3177333250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6815","nonce":3177333250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6813","nonce":3177333250}]},"public_addr":"172.21.15.50:6809/3177333250","cluster_addr":"172.21.15.50:6811/3177333250","heartbeat_back_addr":"172.21.15.50:6815/3177333250","heartbeat_front_addr":"172.21.15.50:6813/3177333250","state":["exists","up"]},{"osd":5,"uuid":"b7d65851-ebe1-45e2-833d-8022446f2698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6809","nonce":4113107737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6811","nonce":4113107737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6815","nonce":4113107737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6813","nonce":4113107737}]},"public_addr":"172.21.15.142:6809/4113107737","cluster_addr":"172.21.15.142:6811/4113107737","heartbeat_back_addr":"172.21.15.142:6815/4113107737","heartbeat_front_addr":"172.21.15.142:6813/4113107737","state":["exists","up"]},{"osd":6,"uuid":"2a3e96d7-d5bf-44ff-9134-29f1420c0e4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6819","nonce":2325162882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6821","nonce":2325162882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6825","nonce":2325162882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6823","nonce":2325162882}]},"public_addr":"172.21.15.46:6819/2325162882","cluster_addr":"172.21.15.46:6821/2325162882","heartbeat_back_addr":"172.21.15.46:6825/2325162882","heartbeat_front_addr":"172.21.15.46:6823/2325162882","state":["exists","up"]},{"osd":7,"uuid":"e53ae885-23b6-46fd-b4b1-c62f31a6921f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6817","nonce":820310228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6819","nonce":820310228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6823","nonce":820310228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6821","nonce":820310228}]},"public_addr":"172.21.15.50:6817/820310228","cluster_addr":"172.21.15.50:6819/820310228","heartbeat_back_addr":"172.21.15.50:6823/820310228","heartbeat_front_addr":"172.21.15.50:6821/820310228","state":["exists","up"]},{"osd":8,"uuid":"356e2692-16a9-4045-b58f-c8f9cb5d7241","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6817","nonce":1824918711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6819","nonce":1824918711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6823","nonce":1824918711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6821","nonce":1824918711}]},"public_addr":"172.21.15.142:6817/1824918711","cluster_addr":"172.21.15.142:6819/1824918711","heartbeat_back_addr":"172.21.15.142:6823/1824918711","heartbeat_front_addr":"172.21.15.142:6821/1824918711","state":["exists","up"]},{"osd":9,"uuid":"8e55570e-77e5-4892-8b9f-77dcd03a4eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6827","nonce":3601947736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6829","nonce":3601947736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6833","nonce":3601947736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6831","nonce":3601947736}]},"public_addr":"172.21.15.46:6827/3601947736","cluster_addr":"172.21.15.46:6829/3601947736","heartbeat_back_addr":"172.21.15.46:6833/3601947736","heartbeat_front_addr":"172.21.15.46:6831/3601947736","state":["exists","up"]},{"osd":10,"uuid":"366389fa-fb9a-408b-b5c2-343c054c5a64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6825","nonce":3135765006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6827","nonce":3135765006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6831","nonce":3135765006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6829","nonce":3135765006}]},"public_addr":"172.21.15.50:6825/3135765006","cluster_addr":"172.21.15.50:6827/3135765006","heartbeat_back_addr":"172.21.15.50:6831/3135765006","heartbeat_front_addr":"172.21.15.50:6829/3135765006","state":["exists","up"]},{"osd":11,"uuid":"7b6e3d93-59e1-4853-b4a2-261b728c47a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6825","nonce":1482712612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6827","nonce":1482712612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6831","nonce":1482712612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6829","nonce":1482712612}]},"public_addr":"172.21.15.142:6825/1482712612","cluster_addr":"172.21.15.142:6827/1482712612","heartbeat_back_addr":"172.21.15.142:6831/1482712612","heartbeat_front_addr":"172.21.15.142:6829/1482712612","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:08.082015+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:32.634928+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:55.978864+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:21.796863+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:46.862518+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:11.440432+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:37.764777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:03.374958+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:28.714052+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:55.761167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:22.330951+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:47.839334+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/2465242088":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6801/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6800/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/1381997090":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/3507268217":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2570024356":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/2691061938":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2775954801":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6800/2896098005":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6801/2896098005":"2024-04-27T21:25:02.078730+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:32:41.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.226+0000 7f677aaf1640 1 -- 172.21.15.46:0/2984770840 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f67640a3bb0 msgr2=0x7f67640a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:41.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.226+0000 7f677aaf1640 1 --2- 172.21.15.46:0/2984770840 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f67640a3bb0 0x7f67640a6070 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6768009bd0 tx=0x7f6768005f90 comp rx=0 tx=0).stop 2024-04-26T21:32:41.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.226+0000 7f677aaf1640 1 -- 172.21.15.46:0/2984770840 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6774108650 msgr2=0x7f67741a0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:41.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.226+0000 7f677aaf1640 1 --2- 172.21.15.46:0/2984770840 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6774108650 0x7f67741a0100 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f676c031a20 tx=0x7f676c0316f0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.226+0000 7f677aaf1640 1 -- 172.21.15.46:0/2984770840 shutdown_connections 2024-04-26T21:32:41.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.226+0000 7f677aaf1640 1 --2- 172.21.15.46:0/2984770840 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f67640a3bb0 0x7f67640a6070 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.226+0000 7f677aaf1640 1 --2- 172.21.15.46:0/2984770840 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6774108650 0x7f67741a0100 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.226+0000 7f677aaf1640 1 --2- 172.21.15.46:0/2984770840 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6774105d20 0x7f6774199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.226+0000 7f677aaf1640 1 --2- 172.21.15.46:0/2984770840 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f67741033f0 0x7f6774198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.226+0000 7f677aaf1640 1 -- 172.21.15.46:0/2984770840 >> 172.21.15.46:0/2984770840 conn(0x7f67740fcde0 msgr2=0x7f6774103d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:41.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.226+0000 7f677aaf1640 1 -- 172.21.15.46:0/2984770840 shutdown_connections 2024-04-26T21:32:41.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.226+0000 7f677aaf1640 1 -- 172.21.15.46:0/2984770840 wait complete. 2024-04-26T21:32:41.239 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T21:27:57.935988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T21:32:41.239 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T21:32:41.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.348+0000 7fb03376a640 1 -- 172.21.15.46:0/1772922597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c1034b0 msgr2=0x7fb02c1058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:41.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.348+0000 7fb03376a640 1 --2- 172.21.15.46:0/1772922597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c1034b0 0x7fb02c1058a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb028009900 tx=0x7fb02802a410 comp rx=0 tx=0).stop 2024-04-26T21:32:41.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.348+0000 7fb03376a640 1 -- 172.21.15.46:0/1772922597 shutdown_connections 2024-04-26T21:32:41.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.348+0000 7fb03376a640 1 --2- 172.21.15.46:0/1772922597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb02c108710 0x7fb02c10ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.348+0000 7fb03376a640 1 --2- 172.21.15.46:0/1772922597 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb02c105de0 0x7fb02c1081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.348+0000 7fb03376a640 1 --2- 172.21.15.46:0/1772922597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c1034b0 0x7fb02c1058a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.348+0000 7fb03376a640 1 -- 172.21.15.46:0/1772922597 >> 172.21.15.46:0/1772922597 conn(0x7fb02c0fcde0 msgr2=0x7fb02c0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:41.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.348+0000 7fb03376a640 1 -- 172.21.15.46:0/1772922597 shutdown_connections 2024-04-26T21:32:41.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.348+0000 7fb03376a640 1 -- 172.21.15.46:0/1772922597 wait complete. 2024-04-26T21:32:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.348+0000 7fb03376a640 1 Processor -- start 2024-04-26T21:32:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.348+0000 7fb03376a640 1 -- start start 2024-04-26T21:32:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.349+0000 7fb03376a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb02c1034b0 0x7fb02c198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.349+0000 7fb03376a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c105de0 0x7fb02c199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.349+0000 7fb03376a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb02c108710 0x7fb02c1a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.349+0000 7fb03376a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb02c10e4d0 con 0x7fb02c105de0 2024-04-26T21:32:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.349+0000 7fb03376a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb02c10e350 con 0x7fb02c1034b0 2024-04-26T21:32:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.349+0000 7fb03376a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb02c10e650 con 0x7fb02c108710 2024-04-26T21:32:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.349+0000 7fb030d05640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c105de0 0x7fb02c199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.349+0000 7fb030d05640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c105de0 0x7fb02c199170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42096/0 (socket says 172.21.15.46:42096) 2024-04-26T21:32:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.349+0000 7fb030d05640 1 -- 172.21.15.46:0/3434702778 learned_addr learned my addr 172.21.15.46:0/3434702778 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.349+0000 7fb031506640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb02c1034b0 0x7fb02c198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.349+0000 7fb031d07640 1 --2- 172.21.15.46:0/3434702778 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb02c108710 0x7fb02c1a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.349+0000 7fb030d05640 1 -- 172.21.15.46:0/3434702778 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb02c108710 msgr2=0x7fb02c1a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.349+0000 7fb030d05640 1 --2- 172.21.15.46:0/3434702778 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb02c108710 0x7fb02c1a01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.349+0000 7fb030d05640 1 -- 172.21.15.46:0/3434702778 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb02c1034b0 msgr2=0x7fb02c198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.349+0000 7fb030d05640 1 --2- 172.21.15.46:0/3434702778 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb02c1034b0 0x7fb02c198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.349+0000 7fb030d05640 1 -- 172.21.15.46:0/3434702778 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb02c1a0840 con 0x7fb02c105de0 2024-04-26T21:32:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.349+0000 7fb031506640 1 --2- 172.21.15.46:0/3434702778 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb02c1034b0 0x7fb02c198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:32:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.349+0000 7fb031d07640 1 --2- 172.21.15.46:0/3434702778 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb02c108710 0x7fb02c1a01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:32:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.349+0000 7fb030d05640 1 --2- 172.21.15.46:0/3434702778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c105de0 0x7fb02c199170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb01c00e9f0 tx=0x7fb01c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.349+0000 7fb0227fc640 1 -- 172.21.15.46:0/3434702778 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb01c00ccc0 con 0x7fb02c105de0 2024-04-26T21:32:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.349+0000 7fb03376a640 1 -- 172.21.15.46:0/3434702778 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb02c1a0b30 con 0x7fb02c105de0 2024-04-26T21:32:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.349+0000 7fb0227fc640 1 -- 172.21.15.46:0/3434702778 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb01c0054d0 con 0x7fb02c105de0 2024-04-26T21:32:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.349+0000 7fb0227fc640 1 -- 172.21.15.46:0/3434702778 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb01c010430 con 0x7fb02c105de0 2024-04-26T21:32:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.350+0000 7fb03376a640 1 -- 172.21.15.46:0/3434702778 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb02c1a1080 con 0x7fb02c105de0 2024-04-26T21:32:41.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.350+0000 7fb03376a640 1 -- 172.21.15.46:0/3434702778 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb02c0fe8d0 con 0x7fb02c105de0 2024-04-26T21:32:41.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.352+0000 7fb0227fc640 1 -- 172.21.15.46:0/3434702778 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb01c0046c0 con 0x7fb02c105de0 2024-04-26T21:32:41.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.352+0000 7fb0227fc640 1 --2- 172.21.15.46:0/3434702778 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb0180a3bb0 0x7fb0180a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:41.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.352+0000 7fb031506640 1 --2- 172.21.15.46:0/3434702778 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb0180a3bb0 0x7fb0180a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:41.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.353+0000 7fb0227fc640 1 -- 172.21.15.46:0/3434702778 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb01c0d0760 con 0x7fb02c105de0 2024-04-26T21:32:41.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.353+0000 7fb031506640 1 --2- 172.21.15.46:0/3434702778 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb0180a3bb0 0x7fb0180a6070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb028009230 tx=0x7fb0280314a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:41.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.360+0000 7fb0227fc640 1 -- 172.21.15.46:0/3434702778 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb01c09d540 con 0x7fb02c105de0 2024-04-26T21:32:41.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:41 smithi142 ceph-mon[32944]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:41.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:41 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/1971863550' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:41.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:41 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:41.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:41 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:41.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:41 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:41.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:41 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:41.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:41 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/2984770840' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:32:41.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.548+0000 7fb03376a640 1 -- 172.21.15.46:0/3434702778 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb02c02dad0 con 0x7fb02c105de0 2024-04-26T21:32:41.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.548+0000 7fb0227fc640 1 -- 172.21.15.46:0/3434702778 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fb01c09ce40 con 0x7fb02c105de0 2024-04-26T21:32:41.549 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-26T21:32:41.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.550+0000 7fb03376a640 1 -- 172.21.15.46:0/3434702778 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb0180a3bb0 msgr2=0x7fb0180a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:41.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.550+0000 7fb03376a640 1 --2- 172.21.15.46:0/3434702778 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb0180a3bb0 0x7fb0180a6070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb028009230 tx=0x7fb0280314a0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.550+0000 7fb03376a640 1 -- 172.21.15.46:0/3434702778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c105de0 msgr2=0x7fb02c199170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:41.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.550+0000 7fb03376a640 1 --2- 172.21.15.46:0/3434702778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c105de0 0x7fb02c199170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb01c00e9f0 tx=0x7fb01c00eeb0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.550+0000 7fb03376a640 1 -- 172.21.15.46:0/3434702778 shutdown_connections 2024-04-26T21:32:41.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.550+0000 7fb03376a640 1 --2- 172.21.15.46:0/3434702778 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb0180a3bb0 0x7fb0180a6070 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.550+0000 7fb03376a640 1 --2- 172.21.15.46:0/3434702778 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb02c108710 0x7fb02c1a01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.550+0000 7fb03376a640 1 --2- 172.21.15.46:0/3434702778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c105de0 0x7fb02c199170 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.550+0000 7fb03376a640 1 --2- 172.21.15.46:0/3434702778 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb02c1034b0 0x7fb02c198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.550+0000 7fb03376a640 1 -- 172.21.15.46:0/3434702778 >> 172.21.15.46:0/3434702778 conn(0x7fb02c0fcde0 msgr2=0x7fb02c103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:41.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.550+0000 7fb03376a640 1 -- 172.21.15.46:0/3434702778 shutdown_connections 2024-04-26T21:32:41.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.550+0000 7fb03376a640 1 -- 172.21.15.46:0/3434702778 wait complete. 2024-04-26T21:32:41.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:41 smithi046 ceph-mon[29201]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:41.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:41 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1971863550' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:41.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:41 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:41.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:41 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:41.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:41 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:41.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:41 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:41.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:41 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2984770840' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:32:41.563 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-26T21:32:41.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:41 smithi050 ceph-mon[32753]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:41.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:41 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/1971863550' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:41.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:41 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:41.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:41 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:41.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:41 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:41.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:41 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:41.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:41 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/2984770840' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:32:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.666+0000 7f6af639d640 1 -- 172.21.15.46:0/1179718216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af01033f0 msgr2=0x7f6af01057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.666+0000 7f6af639d640 1 --2- 172.21.15.46:0/1179718216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af01033f0 0x7f6af01057e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0009900 tx=0x7f6ae002a410 comp rx=0 tx=0).stop 2024-04-26T21:32:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.666+0000 7f6af639d640 1 -- 172.21.15.46:0/1179718216 shutdown_connections 2024-04-26T21:32:41.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.666+0000 7f6af639d640 1 --2- 172.21.15.46:0/1179718216 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6af0108650 0x7f6af010aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.666+0000 7f6af639d640 1 --2- 172.21.15.46:0/1179718216 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6af0105d20 0x7f6af0108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.666+0000 7f6af639d640 1 --2- 172.21.15.46:0/1179718216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af01033f0 0x7f6af01057e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.666+0000 7f6af639d640 1 -- 172.21.15.46:0/1179718216 >> 172.21.15.46:0/1179718216 conn(0x7f6af00fce20 msgr2=0x7f6af00ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.666+0000 7f6af639d640 1 -- 172.21.15.46:0/1179718216 shutdown_connections 2024-04-26T21:32:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.666+0000 7f6af639d640 1 -- 172.21.15.46:0/1179718216 wait complete. 2024-04-26T21:32:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.667+0000 7f6af639d640 1 Processor -- start 2024-04-26T21:32:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.667+0000 7f6af639d640 1 -- start start 2024-04-26T21:32:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.667+0000 7f6af639d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6af01033f0 0x7f6af0198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.667+0000 7f6af639d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6af0105d20 0x7f6af0199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.667+0000 7f6af639d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af0108650 0x7f6af01a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.667+0000 7f6af639d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6af010e4c0 con 0x7f6af0108650 2024-04-26T21:32:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.667+0000 7f6af639d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6af010e340 con 0x7f6af0105d20 2024-04-26T21:32:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.667+0000 7f6af639d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6af010e640 con 0x7f6af01033f0 2024-04-26T21:32:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.667+0000 7f6af493a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af0108650 0x7f6af01a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:41.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.667+0000 7f6af493a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af0108650 0x7f6af01a0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42106/0 (socket says 172.21.15.46:42106) 2024-04-26T21:32:41.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.667+0000 7f6af493a640 1 -- 172.21.15.46:0/2456671007 learned_addr learned my addr 172.21.15.46:0/2456671007 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:41.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.667+0000 7f6aef7fe640 1 --2- 172.21.15.46:0/2456671007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6af0105d20 0x7f6af0199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:41.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.667+0000 7f6aeffff640 1 --2- 172.21.15.46:0/2456671007 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6af01033f0 0x7f6af0198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:41.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.667+0000 7f6af493a640 1 -- 172.21.15.46:0/2456671007 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6af01033f0 msgr2=0x7f6af0198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:41.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.667+0000 7f6af493a640 1 --2- 172.21.15.46:0/2456671007 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6af01033f0 0x7f6af0198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.667+0000 7f6af493a640 1 -- 172.21.15.46:0/2456671007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6af0105d20 msgr2=0x7f6af0199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:41.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.667+0000 7f6af493a640 1 --2- 172.21.15.46:0/2456671007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6af0105d20 0x7f6af0199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.667+0000 7f6af493a640 1 -- 172.21.15.46:0/2456671007 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6af01a07f0 con 0x7f6af0108650 2024-04-26T21:32:41.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.668+0000 7f6af493a640 1 --2- 172.21.15.46:0/2456671007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af0108650 0x7f6af01a0180 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6ae800eb10 tx=0x7f6ae800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:41.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.668+0000 7f6aed7fa640 1 -- 172.21.15.46:0/2456671007 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8004b20 con 0x7f6af0108650 2024-04-26T21:32:41.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.668+0000 7f6aed7fa640 1 -- 172.21.15.46:0/2456671007 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ae8004cc0 con 0x7f6af0108650 2024-04-26T21:32:41.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.668+0000 7f6aed7fa640 1 -- 172.21.15.46:0/2456671007 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8010670 con 0x7f6af0108650 2024-04-26T21:32:41.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.668+0000 7f6af639d640 1 -- 172.21.15.46:0/2456671007 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6af01a0ae0 con 0x7f6af0108650 2024-04-26T21:32:41.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.668+0000 7f6af639d640 1 -- 172.21.15.46:0/2456671007 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6af01a1030 con 0x7f6af0108650 2024-04-26T21:32:41.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.669+0000 7f6af639d640 1 -- 172.21.15.46:0/2456671007 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6af00fe910 con 0x7f6af0108650 2024-04-26T21:32:41.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.670+0000 7f6aed7fa640 1 -- 172.21.15.46:0/2456671007 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6ae8005020 con 0x7f6af0108650 2024-04-26T21:32:41.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.671+0000 7f6aed7fa640 1 --2- 172.21.15.46:0/2456671007 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6ad80a3bb0 0x7f6ad80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:41.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.671+0000 7f6aeffff640 1 --2- 172.21.15.46:0/2456671007 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6ad80a3bb0 0x7f6ad80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:41.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.671+0000 7f6aed7fa640 1 -- 172.21.15.46:0/2456671007 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6ae80d0480 con 0x7f6af0108650 2024-04-26T21:32:41.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.672+0000 7f6aeffff640 1 --2- 172.21.15.46:0/2456671007 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6ad80a3bb0 0x7f6ad80a6070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0000c00 tx=0x7f6ae00314a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:41.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.676+0000 7f6aed7fa640 1 -- 172.21.15.46:0/2456671007 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6ae809d300 con 0x7f6af0108650 2024-04-26T21:32:41.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.870+0000 7f6af639d640 1 -- 172.21.15.46:0/2456671007 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f6af002dd20 con 0x7f6af0108650 2024-04-26T21:32:41.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.871+0000 7f6aed7fa640 1 -- 172.21.15.46:0/2456671007 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7f6ae809cc00 con 0x7f6af0108650 2024-04-26T21:32:41.872 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:32:41.872 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-04-26T21:32:41.872 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-04-26T21:32:41.872 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-04-26T21:32:41.872 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-04-26T21:32:41.872 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-04-26T21:32:41.872 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-04-26T21:32:41.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.873+0000 7f6af639d640 1 -- 172.21.15.46:0/2456671007 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6ad80a3bb0 msgr2=0x7f6ad80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:41.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.873+0000 7f6af639d640 1 --2- 172.21.15.46:0/2456671007 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6ad80a3bb0 0x7f6ad80a6070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0000c00 tx=0x7f6ae00314a0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.873+0000 7f6af639d640 1 -- 172.21.15.46:0/2456671007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af0108650 msgr2=0x7f6af01a0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:41.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.873+0000 7f6af639d640 1 --2- 172.21.15.46:0/2456671007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af0108650 0x7f6af01a0180 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6ae800eb10 tx=0x7f6ae800c2d0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.873+0000 7f6af639d640 1 -- 172.21.15.46:0/2456671007 shutdown_connections 2024-04-26T21:32:41.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.873+0000 7f6af639d640 1 --2- 172.21.15.46:0/2456671007 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6ad80a3bb0 0x7f6ad80a6070 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.873+0000 7f6af639d640 1 --2- 172.21.15.46:0/2456671007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6af0108650 0x7f6af01a0180 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.873+0000 7f6af639d640 1 --2- 172.21.15.46:0/2456671007 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6af0105d20 0x7f6af0199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.873+0000 7f6af639d640 1 --2- 172.21.15.46:0/2456671007 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6af01033f0 0x7f6af0198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.873+0000 7f6af639d640 1 -- 172.21.15.46:0/2456671007 >> 172.21.15.46:0/2456671007 conn(0x7f6af00fce20 msgr2=0x7f6af0103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:41.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.873+0000 7f6af639d640 1 -- 172.21.15.46:0/2456671007 shutdown_connections 2024-04-26T21:32:41.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.873+0000 7f6af639d640 1 -- 172.21.15.46:0/2456671007 wait complete. 2024-04-26T21:32:41.885 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-26T21:32:41.885 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-26T21:32:41.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.990+0000 7f648c11c640 1 -- 172.21.15.46:0/2813949110 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6484108670 msgr2=0x7f648410aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.990+0000 7f648c11c640 1 --2- 172.21.15.46:0/2813949110 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6484108670 0x7f648410aac0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f647c009ca0 tx=0x7f647c02a8c0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.990+0000 7f648c11c640 1 -- 172.21.15.46:0/2813949110 shutdown_connections 2024-04-26T21:32:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.990+0000 7f648c11c640 1 --2- 172.21.15.46:0/2813949110 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6484108670 0x7f648410aac0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.990+0000 7f648c11c640 1 --2- 172.21.15.46:0/2813949110 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6484105d40 0x7f6484108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.990+0000 7f648c11c640 1 --2- 172.21.15.46:0/2813949110 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6484103410 0x7f6484105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.990+0000 7f648c11c640 1 -- 172.21.15.46:0/2813949110 >> 172.21.15.46:0/2813949110 conn(0x7f64840fce20 msgr2=0x7f64840ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.990+0000 7f648c11c640 1 -- 172.21.15.46:0/2813949110 shutdown_connections 2024-04-26T21:32:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.991+0000 7f648c11c640 1 -- 172.21.15.46:0/2813949110 wait complete. 2024-04-26T21:32:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.991+0000 7f648c11c640 1 Processor -- start 2024-04-26T21:32:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.991+0000 7f648c11c640 1 -- start start 2024-04-26T21:32:41.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.991+0000 7f648c11c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6484103410 0x7f6484198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:41.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.991+0000 7f648c11c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6484105d40 0x7f6484199070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:41.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.991+0000 7f648c11c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6484108670 0x7f64841a00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:41.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.991+0000 7f648c11c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f648410e4e0 con 0x7f6484108670 2024-04-26T21:32:41.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.991+0000 7f648c11c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f648410e360 con 0x7f6484103410 2024-04-26T21:32:41.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.991+0000 7f648c11c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f648410e660 con 0x7f6484105d40 2024-04-26T21:32:41.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.991+0000 7f648a6b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6484108670 0x7f64841a00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:41.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.991+0000 7f6489eb8640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6484103410 0x7f6484198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:41.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.991+0000 7f648a6b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6484108670 0x7f64841a00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42124/0 (socket says 172.21.15.46:42124) 2024-04-26T21:32:41.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.991+0000 7f6489eb8640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6484103410 0x7f6484198b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.46:53506/0 (socket says 172.21.15.46:53506) 2024-04-26T21:32:41.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.991+0000 7f648a6b9640 1 -- 172.21.15.46:0/3194591969 learned_addr learned my addr 172.21.15.46:0/3194591969 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:41.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.992+0000 7f64896b7640 1 --2- 172.21.15.46:0/3194591969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6484105d40 0x7f6484199070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:41.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.992+0000 7f648a6b9640 1 -- 172.21.15.46:0/3194591969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6484105d40 msgr2=0x7f6484199070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:41.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.992+0000 7f648a6b9640 1 --2- 172.21.15.46:0/3194591969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6484105d40 0x7f6484199070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.992+0000 7f648a6b9640 1 -- 172.21.15.46:0/3194591969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6484103410 msgr2=0x7f6484198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.992+0000 7f648a6b9640 1 --2- 172.21.15.46:0/3194591969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6484103410 0x7f6484198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.992+0000 7f648a6b9640 1 -- 172.21.15.46:0/3194591969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64841a0750 con 0x7f6484108670 2024-04-26T21:32:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.992+0000 7f648a6b9640 1 --2- 172.21.15.46:0/3194591969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6484108670 0x7f64841a00e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f647c0097b0 tx=0x7f647c003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.992+0000 7f647affd640 1 -- 172.21.15.46:0/3194591969 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c004240 con 0x7f6484108670 2024-04-26T21:32:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.992+0000 7f648c11c640 1 -- 172.21.15.46:0/3194591969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64841a09e0 con 0x7f6484108670 2024-04-26T21:32:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.992+0000 7f647affd640 1 -- 172.21.15.46:0/3194591969 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f647c0317c0 con 0x7f6484108670 2024-04-26T21:32:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.992+0000 7f647affd640 1 -- 172.21.15.46:0/3194591969 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c02fd80 con 0x7f6484108670 2024-04-26T21:32:41.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.992+0000 7f648c11c640 1 -- 172.21.15.46:0/3194591969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64841a0f30 con 0x7f6484108670 2024-04-26T21:32:41.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.993+0000 7f648c11c640 1 -- 172.21.15.46:0/3194591969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64840fe910 con 0x7f6484108670 2024-04-26T21:32:41.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.994+0000 7f647affd640 1 -- 172.21.15.46:0/3194591969 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f647c0043e0 con 0x7f6484108670 2024-04-26T21:32:41.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.995+0000 7f647affd640 1 --2- 172.21.15.46:0/3194591969 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f64700a3bb0 0x7f64700a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:41.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.995+0000 7f6489eb8640 1 --2- 172.21.15.46:0/3194591969 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f64700a3bb0 0x7f64700a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:41.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.995+0000 7f647affd640 1 -- 172.21.15.46:0/3194591969 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f647c0f1890 con 0x7f6484108670 2024-04-26T21:32:41.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.995+0000 7f6489eb8640 1 --2- 172.21.15.46:0/3194591969 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f64700a3bb0 0x7f64700a6070 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6480005fd0 tx=0x7f648000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:42.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:41.999+0000 7f647affd640 1 -- 172.21.15.46:0/3194591969 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f647c0be670 con 0x7f6484108670 2024-04-26T21:32:42.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.190+0000 7f648c11c640 1 -- 172.21.15.46:0/3194591969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f648402dcb0 con 0x7f6484108670 2024-04-26T21:32:42.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:42 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:42.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:42 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:42.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:42 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3434702778' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:32:42.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:42 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:42.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:42 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:32:42.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:42 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:42.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:42 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/2456671007' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T21:32:42.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:42 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:32:42.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:42 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3194591969' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T21:32:42.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:42 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:42.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:42 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:42.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:42 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3434702778' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:32:42.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:42 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:42.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:42 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:32:42.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:42 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:42.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:42 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2456671007' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T21:32:42.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:42 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:32:42.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:42 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3194591969' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T21:32:42.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:42 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:42.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:42 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:42.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:42 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3434702778' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:32:42.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:42 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:42.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:42 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:32:42.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:42 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:42.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:42 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/2456671007' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T21:32:42.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:42 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:32:42.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:42 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3194591969' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T21:32:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.823+0000 7f647affd640 1 -- 172.21.15.46:0/3194591969 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7f647c0098b0 con 0x7f6484108670 2024-04-26T21:32:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-04-26T21:32:42.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.825+0000 7f648c11c640 1 -- 172.21.15.46:0/3194591969 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f64700a3bb0 msgr2=0x7f64700a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.825+0000 7f648c11c640 1 --2- 172.21.15.46:0/3194591969 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f64700a3bb0 0x7f64700a6070 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6480005fd0 tx=0x7f648000b390 comp rx=0 tx=0).stop 2024-04-26T21:32:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.825+0000 7f648c11c640 1 -- 172.21.15.46:0/3194591969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6484108670 msgr2=0x7f64841a00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.825+0000 7f648c11c640 1 --2- 172.21.15.46:0/3194591969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6484108670 0x7f64841a00e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f647c0097b0 tx=0x7f647c003c20 comp rx=0 tx=0).stop 2024-04-26T21:32:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.825+0000 7f648c11c640 1 -- 172.21.15.46:0/3194591969 shutdown_connections 2024-04-26T21:32:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.825+0000 7f648c11c640 1 --2- 172.21.15.46:0/3194591969 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f64700a3bb0 0x7f64700a6070 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.825+0000 7f648c11c640 1 --2- 172.21.15.46:0/3194591969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6484108670 0x7f64841a00e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.825+0000 7f648c11c640 1 --2- 172.21.15.46:0/3194591969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6484105d40 0x7f6484199070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.826+0000 7f648c11c640 1 --2- 172.21.15.46:0/3194591969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6484103410 0x7f6484198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.826+0000 7f648c11c640 1 -- 172.21.15.46:0/3194591969 >> 172.21.15.46:0/3194591969 conn(0x7f64840fce20 msgr2=0x7f6484103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.826+0000 7f648c11c640 1 -- 172.21.15.46:0/3194591969 shutdown_connections 2024-04-26T21:32:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.826+0000 7f648c11c640 1 -- 172.21.15.46:0/3194591969 wait complete. 2024-04-26T21:32:42.843 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-26T21:32:42.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.958+0000 7f353c452640 1 -- 172.21.15.46:0/209293884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3534103b50 msgr2=0x7f3534103fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:42.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.958+0000 7f353c452640 1 --2- 172.21.15.46:0/209293884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3534103b50 0x7f3534103fd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3524009900 tx=0x7f352402a410 comp rx=0 tx=0).stop 2024-04-26T21:32:42.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.958+0000 7f353c452640 1 -- 172.21.15.46:0/209293884 shutdown_connections 2024-04-26T21:32:42.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.958+0000 7f353c452640 1 --2- 172.21.15.46:0/209293884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3534104510 0x7f353410ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:42.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.958+0000 7f353c452640 1 --2- 172.21.15.46:0/209293884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3534103b50 0x7f3534103fd0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:42.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.958+0000 7f353c452640 1 --2- 172.21.15.46:0/209293884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3534102960 0x7f3534102d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:42.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.958+0000 7f353c452640 1 -- 172.21.15.46:0/209293884 >> 172.21.15.46:0/209293884 conn(0x7f35340fdef0 msgr2=0x7f3534100310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:42.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.959+0000 7f353c452640 1 -- 172.21.15.46:0/209293884 shutdown_connections 2024-04-26T21:32:42.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.959+0000 7f353c452640 1 -- 172.21.15.46:0/209293884 wait complete. 2024-04-26T21:32:42.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.959+0000 7f353c452640 1 Processor -- start 2024-04-26T21:32:42.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.959+0000 7f353c452640 1 -- start start 2024-04-26T21:32:42.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.959+0000 7f353c452640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3534102960 0x7f3534198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:42.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.959+0000 7f353c452640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3534103b50 0x7f3534199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:42.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.959+0000 7f353c452640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3534104510 0x7f35341a0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:42.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.959+0000 7f353c452640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f353410e490 con 0x7f3534102960 2024-04-26T21:32:42.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.959+0000 7f353c452640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f353410e310 con 0x7f3534104510 2024-04-26T21:32:42.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.959+0000 7f353c452640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f353410e610 con 0x7f3534103b50 2024-04-26T21:32:42.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.959+0000 7f353a1ee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3534102960 0x7f3534198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:42.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.959+0000 7f353a9ef640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3534104510 0x7f35341a0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:42.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.959+0000 7f35399ed640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3534103b50 0x7f3534199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:42.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.959+0000 7f353a9ef640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3534104510 0x7f35341a0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.46:53526/0 (socket says 172.21.15.46:53526) 2024-04-26T21:32:42.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.959+0000 7f353a9ef640 1 -- 172.21.15.46:0/2404404247 learned_addr learned my addr 172.21.15.46:0/2404404247 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:42.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.960+0000 7f35399ed640 1 -- 172.21.15.46:0/2404404247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3534104510 msgr2=0x7f35341a0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:42.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.960+0000 7f35399ed640 1 --2- 172.21.15.46:0/2404404247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3534104510 0x7f35341a0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:42.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.960+0000 7f35399ed640 1 -- 172.21.15.46:0/2404404247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3534102960 msgr2=0x7f3534198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:42.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.960+0000 7f35399ed640 1 --2- 172.21.15.46:0/2404404247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3534102960 0x7f3534198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:42.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.960+0000 7f35399ed640 1 -- 172.21.15.46:0/2404404247 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35341a07e0 con 0x7f3534103b50 2024-04-26T21:32:42.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.960+0000 7f35399ed640 1 --2- 172.21.15.46:0/2404404247 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3534103b50 0x7f3534199100 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3524006010 tx=0x7f35240042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:42.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.960+0000 7f352b7fe640 1 -- 172.21.15.46:0/2404404247 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352403f030 con 0x7f3534103b50 2024-04-26T21:32:42.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.960+0000 7f353c452640 1 -- 172.21.15.46:0/2404404247 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35341a0a70 con 0x7f3534103b50 2024-04-26T21:32:42.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.960+0000 7f352b7fe640 1 -- 172.21.15.46:0/2404404247 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3524031910 con 0x7f3534103b50 2024-04-26T21:32:42.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.960+0000 7f352b7fe640 1 -- 172.21.15.46:0/2404404247 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3524035070 con 0x7f3534103b50 2024-04-26T21:32:42.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.960+0000 7f353c452640 1 -- 172.21.15.46:0/2404404247 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35341a0fb0 con 0x7f3534103b50 2024-04-26T21:32:42.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.961+0000 7f353c452640 1 -- 172.21.15.46:0/2404404247 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3534109ca0 con 0x7f3534103b50 2024-04-26T21:32:42.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.963+0000 7f352b7fe640 1 -- 172.21.15.46:0/2404404247 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3524042030 con 0x7f3534103b50 2024-04-26T21:32:42.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.963+0000 7f352b7fe640 1 --2- 172.21.15.46:0/2404404247 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f35200a3bb0 0x7f35200a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:42.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.963+0000 7f353a1ee640 1 --2- 172.21.15.46:0/2404404247 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f35200a3bb0 0x7f35200a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:42.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.963+0000 7f352b7fe640 1 -- 172.21.15.46:0/2404404247 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7f35240f05d0 con 0x7f3534103b50 2024-04-26T21:32:42.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.964+0000 7f353a1ee640 1 --2- 172.21.15.46:0/2404404247 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f35200a3bb0 0x7f35200a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f35341039b0 tx=0x7f3530005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:42.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:42.968+0000 7f352b7fe640 1 -- 172.21.15.46:0/2404404247 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f35240bcd50 con 0x7f3534103b50 2024-04-26T21:32:43.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.156+0000 7f353c452640 1 -- 172.21.15.46:0/2404404247 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f3534063530 con 0x7f3534103b50 2024-04-26T21:32:43.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.829+0000 7f352b7fe640 1 -- 172.21.15.46:0/2404404247 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v83) ==== 170+0+0 (secure 0 0 0) 0x7f35240c1050 con 0x7f3534103b50 2024-04-26T21:32:43.831 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-04-26T21:32:43.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.833+0000 7f353c452640 1 -- 172.21.15.46:0/2404404247 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f35200a3bb0 msgr2=0x7f35200a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:43.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.833+0000 7f353c452640 1 --2- 172.21.15.46:0/2404404247 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f35200a3bb0 0x7f35200a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f35341039b0 tx=0x7f3530005f90 comp rx=0 tx=0).stop 2024-04-26T21:32:43.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.833+0000 7f353c452640 1 -- 172.21.15.46:0/2404404247 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3534103b50 msgr2=0x7f3534199100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:43.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.833+0000 7f353c452640 1 --2- 172.21.15.46:0/2404404247 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3534103b50 0x7f3534199100 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3524006010 tx=0x7f35240042f0 comp rx=0 tx=0).stop 2024-04-26T21:32:43.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.833+0000 7f353c452640 1 -- 172.21.15.46:0/2404404247 shutdown_connections 2024-04-26T21:32:43.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.833+0000 7f353c452640 1 --2- 172.21.15.46:0/2404404247 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f35200a3bb0 0x7f35200a6070 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:43.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.833+0000 7f353c452640 1 --2- 172.21.15.46:0/2404404247 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3534104510 0x7f35341a0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:43.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.833+0000 7f353c452640 1 --2- 172.21.15.46:0/2404404247 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f3534103b50 0x7f3534199100 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:43.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.833+0000 7f353c452640 1 --2- 172.21.15.46:0/2404404247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3534102960 0x7f3534198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:43.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.833+0000 7f353c452640 1 -- 172.21.15.46:0/2404404247 >> 172.21.15.46:0/2404404247 conn(0x7f35340fdef0 msgr2=0x7f3534106e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:43.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.834+0000 7f353c452640 1 -- 172.21.15.46:0/2404404247 shutdown_connections 2024-04-26T21:32:43.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.834+0000 7f353c452640 1 -- 172.21.15.46:0/2404404247 wait complete. 2024-04-26T21:32:43.854 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-26T21:32:43.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.961+0000 7f4a49c09640 1 -- 172.21.15.46:0/3696788147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a44103b70 msgr2=0x7f4a44103ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:43.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.961+0000 7f4a49c09640 1 --2- 172.21.15.46:0/3696788147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a44103b70 0x7f4a44103ff0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4a38009900 tx=0x7f4a3802a410 comp rx=0 tx=0).stop 2024-04-26T21:32:43.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.962+0000 7f4a49c09640 1 -- 172.21.15.46:0/3696788147 shutdown_connections 2024-04-26T21:32:43.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.962+0000 7f4a49c09640 1 --2- 172.21.15.46:0/3696788147 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a44104530 0x7f4a4410adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:43.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.962+0000 7f4a49c09640 1 --2- 172.21.15.46:0/3696788147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a44103b70 0x7f4a44103ff0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:43.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.962+0000 7f4a49c09640 1 --2- 172.21.15.46:0/3696788147 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4a44102980 0x7f4a44102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.962+0000 7f4a49c09640 1 -- 172.21.15.46:0/3696788147 >> 172.21.15.46:0/3696788147 conn(0x7f4a440fdf10 msgr2=0x7f4a44100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.962+0000 7f4a49c09640 1 -- 172.21.15.46:0/3696788147 shutdown_connections 2024-04-26T21:32:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.962+0000 7f4a49c09640 1 -- 172.21.15.46:0/3696788147 wait complete. 2024-04-26T21:32:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.962+0000 7f4a49c09640 1 Processor -- start 2024-04-26T21:32:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.962+0000 7f4a49c09640 1 -- start start 2024-04-26T21:32:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.962+0000 7f4a49c09640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a44102980 0x7f4a440766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.963+0000 7f4a49c09640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a44103b70 0x7f4a44074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.963+0000 7f4a49c09640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4a44104530 0x7f4a44075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.963+0000 7f4a49c09640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4410e4e0 con 0x7f4a44102980 2024-04-26T21:32:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.963+0000 7f4a49c09640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4410e360 con 0x7f4a44103b70 2024-04-26T21:32:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.963+0000 7f4a49c09640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4410e660 con 0x7f4a44104530 2024-04-26T21:32:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.963+0000 7f4a437fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a44102980 0x7f4a440766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:43.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.963+0000 7f4a437fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a44102980 0x7f4a440766e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42164/0 (socket says 172.21.15.46:42164) 2024-04-26T21:32:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.963+0000 7f4a437fe640 1 -- 172.21.15.46:0/1741673035 learned_addr learned my addr 172.21.15.46:0/1741673035 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.963+0000 7f4a43fff640 1 --2- 172.21.15.46:0/1741673035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4a44104530 0x7f4a44075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.963+0000 7f4a42ffd640 1 --2- 172.21.15.46:0/1741673035 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a44103b70 0x7f4a44074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.963+0000 7f4a437fe640 1 -- 172.21.15.46:0/1741673035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4a44104530 msgr2=0x7f4a44075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.963+0000 7f4a437fe640 1 --2- 172.21.15.46:0/1741673035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4a44104530 0x7f4a44075260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.963+0000 7f4a437fe640 1 -- 172.21.15.46:0/1741673035 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a44103b70 msgr2=0x7f4a44074d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.963+0000 7f4a437fe640 1 --2- 172.21.15.46:0/1741673035 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a44103b70 0x7f4a44074d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.963+0000 7f4a437fe640 1 -- 172.21.15.46:0/1741673035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a44075af0 con 0x7f4a44102980 2024-04-26T21:32:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.963+0000 7f4a43fff640 1 --2- 172.21.15.46:0/1741673035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4a44104530 0x7f4a44075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:32:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.963+0000 7f4a42ffd640 1 --2- 172.21.15.46:0/1741673035 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a44103b70 0x7f4a44074d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:32:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.963+0000 7f4a437fe640 1 --2- 172.21.15.46:0/1741673035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a44102980 0x7f4a440766e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4a3400e9c0 tx=0x7f4a3400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.963+0000 7f4a40ff9640 1 -- 172.21.15.46:0/1741673035 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3400cd80 con 0x7f4a44102980 2024-04-26T21:32:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.964+0000 7f4a40ff9640 1 -- 172.21.15.46:0/1741673035 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a340035f0 con 0x7f4a44102980 2024-04-26T21:32:43.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.964+0000 7f4a49c09640 1 -- 172.21.15.46:0/1741673035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a441a2cd0 con 0x7f4a44102980 2024-04-26T21:32:43.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.964+0000 7f4a40ff9640 1 -- 172.21.15.46:0/1741673035 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a34010690 con 0x7f4a44102980 2024-04-26T21:32:43.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.964+0000 7f4a49c09640 1 -- 172.21.15.46:0/1741673035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a441a31a0 con 0x7f4a44102980 2024-04-26T21:32:43.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.965+0000 7f4a49c09640 1 -- 172.21.15.46:0/1741673035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a44069750 con 0x7f4a44102980 2024-04-26T21:32:43.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.966+0000 7f4a40ff9640 1 -- 172.21.15.46:0/1741673035 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4a340037a0 con 0x7f4a44102980 2024-04-26T21:32:43.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.966+0000 7f4a40ff9640 1 --2- 172.21.15.46:0/1741673035 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4a2c0a3bb0 0x7f4a2c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:43.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.967+0000 7f4a40ff9640 1 -- 172.21.15.46:0/1741673035 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7f4a340d0610 con 0x7f4a44102980 2024-04-26T21:32:43.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.967+0000 7f4a42ffd640 1 --2- 172.21.15.46:0/1741673035 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4a2c0a3bb0 0x7f4a2c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:43.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.967+0000 7f4a42ffd640 1 --2- 172.21.15.46:0/1741673035 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4a2c0a3bb0 0x7f4a2c0a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4a38031500 tx=0x7f4a38005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:43.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:43.971+0000 7f4a40ff9640 1 -- 172.21.15.46:0/1741673035 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4a3409d1a0 con 0x7f4a44102980 2024-04-26T21:32:44.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:43 smithi050 ceph-mon[32753]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:44.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:43 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3194591969' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T21:32:44.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:43 smithi050 ceph-mon[32753]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T21:32:44.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:43 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/2404404247' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T21:32:44.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:43 smithi050 ceph-mon[32753]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T21:32:44.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:43 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:44.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:43 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:44.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:43 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:44.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:43 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:44.121 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:43 smithi046 ceph-mon[29201]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:44.121 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:43 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3194591969' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T21:32:44.121 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:43 smithi046 ceph-mon[29201]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T21:32:44.121 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:43 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2404404247' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T21:32:44.121 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:43 smithi046 ceph-mon[29201]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T21:32:44.121 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:43 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:44.121 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:43 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:44.121 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:43 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:44.121 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:43 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:44.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:44.200+0000 7f4a49c09640 1 -- 172.21.15.46:0/1741673035 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f4a44109d20 con 0x7f4a44102980 2024-04-26T21:32:44.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:43 smithi142 ceph-mon[32944]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:44.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:43 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3194591969' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T21:32:44.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:43 smithi142 ceph-mon[32944]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T21:32:44.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:43 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/2404404247' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T21:32:44.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:43 smithi142 ceph-mon[32944]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T21:32:44.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:43 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:44.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:43 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:44.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:43 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:44.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:43 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:44.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:44.957+0000 7f4a40ff9640 1 -- 172.21.15.46:0/1741673035 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f4a3409caa0 con 0x7f4a44102980 2024-04-26T21:32:44.958 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 2 and data pool 3 2024-04-26T21:32:44.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:44.960+0000 7f4a49c09640 1 -- 172.21.15.46:0/1741673035 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4a2c0a3bb0 msgr2=0x7f4a2c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:44.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:44.960+0000 7f4a49c09640 1 --2- 172.21.15.46:0/1741673035 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4a2c0a3bb0 0x7f4a2c0a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4a38031500 tx=0x7f4a38005a20 comp rx=0 tx=0).stop 2024-04-26T21:32:44.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:44.960+0000 7f4a49c09640 1 -- 172.21.15.46:0/1741673035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a44102980 msgr2=0x7f4a440766e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:44.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:44.960+0000 7f4a49c09640 1 --2- 172.21.15.46:0/1741673035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a44102980 0x7f4a440766e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4a3400e9c0 tx=0x7f4a3400ee80 comp rx=0 tx=0).stop 2024-04-26T21:32:44.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:44.961+0000 7f4a49c09640 1 -- 172.21.15.46:0/1741673035 shutdown_connections 2024-04-26T21:32:44.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:44.961+0000 7f4a49c09640 1 --2- 172.21.15.46:0/1741673035 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4a2c0a3bb0 0x7f4a2c0a6070 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:44.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:44.961+0000 7f4a49c09640 1 --2- 172.21.15.46:0/1741673035 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4a44104530 0x7f4a44075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:44.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:44.961+0000 7f4a49c09640 1 --2- 172.21.15.46:0/1741673035 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4a44103b70 0x7f4a44074d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:44.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:44.961+0000 7f4a49c09640 1 --2- 172.21.15.46:0/1741673035 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4a44102980 0x7f4a440766e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:44.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:44.961+0000 7f4a49c09640 1 -- 172.21.15.46:0/1741673035 >> 172.21.15.46:0/1741673035 conn(0x7f4a440fdf10 msgr2=0x7f4a44106ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:44.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:44.961+0000 7f4a49c09640 1 -- 172.21.15.46:0/1741673035 shutdown_connections 2024-04-26T21:32:44.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:44.961+0000 7f4a49c09640 1 -- 172.21.15.46:0/1741673035 wait complete. 2024-04-26T21:32:44.977 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:32:45.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.081+0000 7f5c8902c640 1 -- 172.21.15.46:0/4240640680 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c84101870 msgr2=0x7f5c84101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:45.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.081+0000 7f5c8902c640 1 --2- 172.21.15.46:0/4240640680 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c84101870 0x7f5c84101c90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5c74009900 tx=0x7f5c7402a410 comp rx=0 tx=0).stop 2024-04-26T21:32:45.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.081+0000 7f5c8902c640 1 -- 172.21.15.46:0/4240640680 shutdown_connections 2024-04-26T21:32:45.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.081+0000 7f5c8902c640 1 --2- 172.21.15.46:0/4240640680 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c84103420 0x7f5c84109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:45.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.081+0000 7f5c8902c640 1 --2- 172.21.15.46:0/4240640680 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c84102a60 0x7f5c84102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:45.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.081+0000 7f5c8902c640 1 --2- 172.21.15.46:0/4240640680 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c84101870 0x7f5c84101c90 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:45.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.081+0000 7f5c8902c640 1 -- 172.21.15.46:0/4240640680 >> 172.21.15.46:0/4240640680 conn(0x7f5c840fcde0 msgr2=0x7f5c840ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:45.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.081+0000 7f5c8902c640 1 -- 172.21.15.46:0/4240640680 shutdown_connections 2024-04-26T21:32:45.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.081+0000 7f5c8902c640 1 -- 172.21.15.46:0/4240640680 wait complete. 2024-04-26T21:32:45.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.082+0000 7f5c8902c640 1 Processor -- start 2024-04-26T21:32:45.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.082+0000 7f5c8902c640 1 -- start start 2024-04-26T21:32:45.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.082+0000 7f5c8902c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c84101870 0x7f5c8406a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:45.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.082+0000 7f5c8902c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c84102a60 0x7f5c8419d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:45.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.082+0000 7f5c8902c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c84103420 0x7f5c8419ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:45.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.082+0000 7f5c8902c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8410d3f0 con 0x7f5c84103420 2024-04-26T21:32:45.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.082+0000 7f5c8902c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8410d270 con 0x7f5c84102a60 2024-04-26T21:32:45.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.082+0000 7f5c8902c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8410d570 con 0x7f5c84101870 2024-04-26T21:32:45.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.083+0000 7f5c8359e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c84103420 0x7f5c8419ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:45.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.083+0000 7f5c8359e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c84103420 0x7f5c8419ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42190/0 (socket says 172.21.15.46:42190) 2024-04-26T21:32:45.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.083+0000 7f5c8359e640 1 -- 172.21.15.46:0/2748612044 learned_addr learned my addr 172.21.15.46:0/2748612044 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:45.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.083+0000 7f5c7a59c640 1 --2- 172.21.15.46:0/2748612044 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c84102a60 0x7f5c8419d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:45.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.083+0000 7f5c82d9d640 1 --2- 172.21.15.46:0/2748612044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c84101870 0x7f5c8406a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:45.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.083+0000 7f5c8359e640 1 -- 172.21.15.46:0/2748612044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c84101870 msgr2=0x7f5c8406a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:45.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.083+0000 7f5c8359e640 1 --2- 172.21.15.46:0/2748612044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c84101870 0x7f5c8406a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:45.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.083+0000 7f5c8359e640 1 -- 172.21.15.46:0/2748612044 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c84102a60 msgr2=0x7f5c8419d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:45.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.083+0000 7f5c8359e640 1 --2- 172.21.15.46:0/2748612044 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c84102a60 0x7f5c8419d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:45.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.083+0000 7f5c8359e640 1 -- 172.21.15.46:0/2748612044 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c841a08a0 con 0x7f5c84103420 2024-04-26T21:32:45.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.083+0000 7f5c8359e640 1 --2- 172.21.15.46:0/2748612044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c84103420 0x7f5c8419ffe0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c00eb10 tx=0x7f5c7c00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:45.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.083+0000 7f5c80d99640 1 -- 172.21.15.46:0/2748612044 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c00ce20 con 0x7f5c84103420 2024-04-26T21:32:45.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.083+0000 7f5c80d99640 1 -- 172.21.15.46:0/2748612044 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c7c005020 con 0x7f5c84103420 2024-04-26T21:32:45.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.083+0000 7f5c8902c640 1 -- 172.21.15.46:0/2748612044 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c841a0b90 con 0x7f5c84103420 2024-04-26T21:32:45.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.083+0000 7f5c80d99640 1 -- 172.21.15.46:0/2748612044 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c010690 con 0x7f5c84103420 2024-04-26T21:32:45.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.083+0000 7f5c8902c640 1 -- 172.21.15.46:0/2748612044 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c841a1110 con 0x7f5c84103420 2024-04-26T21:32:45.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.084+0000 7f5c8902c640 1 -- 172.21.15.46:0/2748612044 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c84108c00 con 0x7f5c84103420 2024-04-26T21:32:45.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.085+0000 7f5c80d99640 1 -- 172.21.15.46:0/2748612044 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5c7c005510 con 0x7f5c84103420 2024-04-26T21:32:45.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.086+0000 7f5c80d99640 1 --2- 172.21.15.46:0/2748612044 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5c700a3bb0 0x7f5c700a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:45.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.086+0000 7f5c82d9d640 1 --2- 172.21.15.46:0/2748612044 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5c700a3bb0 0x7f5c700a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:45.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.087+0000 7f5c80d99640 1 -- 172.21.15.46:0/2748612044 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f5c7c0d0b50 con 0x7f5c84103420 2024-04-26T21:32:45.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.087+0000 7f5c82d9d640 1 --2- 172.21.15.46:0/2748612044 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5c700a3bb0 0x7f5c700a6070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5c74000c00 tx=0x7f5c740314a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:45.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.090+0000 7f5c80d99640 1 -- 172.21.15.46:0/2748612044 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5c7c054060 con 0x7f5c84103420 2024-04-26T21:32:45.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:44 smithi050 ceph-mon[32753]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T21:32:45.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:44 smithi050 ceph-mon[32753]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T21:32:45.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:44 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:45.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:44 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:45.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:44 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:45.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:44 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:32:45.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:44 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:45.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:44 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/1741673035' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T21:32:45.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:44 smithi050 ceph-mon[32753]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T21:32:45.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.277+0000 7f5c8902c640 1 -- 172.21.15.46:0/2748612044 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5c8402dcd0 con 0x7f5c84103420 2024-04-26T21:32:45.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.279+0000 7f5c80d99640 1 -- 172.21.15.46:0/2748612044 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22210 (secure 0 0 0) 0x7f5c7c016330 con 0x7f5c84103420 2024-04-26T21:32:45.280 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:32:45.280 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":84,"fsid":"36c7013a-0413-11ef-bc93-c7b262605968","created":"2024-04-26T21:24:30.481156+0000","modified":"2024-04-26T21:32:44.201585+0000","last_up_change":"2024-04-26T21:31:50.100491+0000","last_in_change":"2024-04-26T21:31:29.643428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:27:57.935988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:32:42.195211+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:32:43.162114+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f53c8702-e4da-435f-a9dc-3911d0d36ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6803","nonce":2762665125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6805","nonce":2762665125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6809","nonce":2762665125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6807","nonce":2762665125}]},"public_addr":"172.21.15.46:6803/2762665125","cluster_addr":"172.21.15.46:6805/2762665125","heartbeat_back_addr":"172.21.15.46:6809/2762665125","heartbeat_front_addr":"172.21.15.46:6807/2762665125","state":["exists","up"]},{"osd":1,"uuid":"fc4d6f52-fc8a-4efd-80c9-c092b57e7f42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6801","nonce":1628008970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6803","nonce":1628008970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6807","nonce":1628008970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6805","nonce":1628008970}]},"public_addr":"172.21.15.50:6801/1628008970","cluster_addr":"172.21.15.50:6803/1628008970","heartbeat_back_addr":"172.21.15.50:6807/1628008970","heartbeat_front_addr":"172.21.15.50:6805/1628008970","state":["exists","up"]},{"osd":2,"uuid":"26d1baa1-4ea1-4b7e-9fcf-4f8bbf7329c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6801","nonce":1513279137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6803","nonce":1513279137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6807","nonce":1513279137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6805","nonce":1513279137}]},"public_addr":"172.21.15.142:6801/1513279137","cluster_addr":"172.21.15.142:6803/1513279137","heartbeat_back_addr":"172.21.15.142:6807/1513279137","heartbeat_front_addr":"172.21.15.142:6805/1513279137","state":["exists","up"]},{"osd":3,"uuid":"34d2401f-5fcc-491c-8fcf-c3c492611f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6811","nonce":2435181760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6813","nonce":2435181760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6817","nonce":2435181760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6815","nonce":2435181760}]},"public_addr":"172.21.15.46:6811/2435181760","cluster_addr":"172.21.15.46:6813/2435181760","heartbeat_back_addr":"172.21.15.46:6817/2435181760","heartbeat_front_addr":"172.21.15.46:6815/2435181760","state":["exists","up"]},{"osd":4,"uuid":"986bb709-e4cf-400d-982e-c3f5ac413e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6809","nonce":3177333250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6811","nonce":3177333250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6815","nonce":3177333250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6813","nonce":3177333250}]},"public_addr":"172.21.15.50:6809/3177333250","cluster_addr":"172.21.15.50:6811/3177333250","heartbeat_back_addr":"172.21.15.50:6815/3177333250","heartbeat_front_addr":"172.21.15.50:6813/3177333250","state":["exists","up"]},{"osd":5,"uuid":"b7d65851-ebe1-45e2-833d-8022446f2698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6809","nonce":4113107737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6811","nonce":4113107737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6815","nonce":4113107737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6813","nonce":4113107737}]},"public_addr":"172.21.15.142:6809/4113107737","cluster_addr":"172.21.15.142:6811/4113107737","heartbeat_back_addr":"172.21.15.142:6815/4113107737","heartbeat_front_addr":"172.21.15.142:6813/4113107737","state":["exists","up"]},{"osd":6,"uuid":"2a3e96d7-d5bf-44ff-9134-29f1420c0e4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6819","nonce":2325162882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6821","nonce":2325162882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6825","nonce":2325162882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6823","nonce":2325162882}]},"public_addr":"172.21.15.46:6819/2325162882","cluster_addr":"172.21.15.46:6821/2325162882","heartbeat_back_addr":"172.21.15.46:6825/2325162882","heartbeat_front_addr":"172.21.15.46:6823/2325162882","state":["exists","up"]},{"osd":7,"uuid":"e53ae885-23b6-46fd-b4b1-c62f31a6921f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6817","nonce":820310228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6819","nonce":820310228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6823","nonce":820310228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6821","nonce":820310228}]},"public_addr":"172.21.15.50:6817/820310228","cluster_addr":"172.21.15.50:6819/820310228","heartbeat_back_addr":"172.21.15.50:6823/820310228","heartbeat_front_addr":"172.21.15.50:6821/820310228","state":["exists","up"]},{"osd":8,"uuid":"356e2692-16a9-4045-b58f-c8f9cb5d7241","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6817","nonce":1824918711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6819","nonce":1824918711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6823","nonce":1824918711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6821","nonce":1824918711}]},"public_addr":"172.21.15.142:6817/1824918711","cluster_addr":"172.21.15.142:6819/1824918711","heartbeat_back_addr":"172.21.15.142:6823/1824918711","heartbeat_front_addr":"172.21.15.142:6821/1824918711","state":["exists","up"]},{"osd":9,"uuid":"8e55570e-77e5-4892-8b9f-77dcd03a4eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6827","nonce":3601947736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6829","nonce":3601947736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6833","nonce":3601947736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6831","nonce":3601947736}]},"public_addr":"172.21.15.46:6827/3601947736","cluster_addr":"172.21.15.46:6829/3601947736","heartbeat_back_addr":"172.21.15.46:6833/3601947736","heartbeat_front_addr":"172.21.15.46:6831/3601947736","state":["exists","up"]},{"osd":10,"uuid":"366389fa-fb9a-408b-b5c2-343c054c5a64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6825","nonce":3135765006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6827","nonce":3135765006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6831","nonce":3135765006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6829","nonce":3135765006}]},"public_addr":"172.21.15.50:6825/3135765006","cluster_addr":"172.21.15.50:6827/3135765006","heartbeat_back_addr":"172.21.15.50:6831/3135765006","heartbeat_front_addr":"172.21.15.50:6829/3135765006","state":["exists","up"]},{"osd":11,"uuid":"7b6e3d93-59e1-4853-b4a2-261b728c47a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6825","nonce":1482712612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6827","nonce":1482712612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6831","nonce":1482712612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6829","nonce":1482712612}]},"public_addr":"172.21.15.142:6825/1482712612","cluster_addr":"172.21.15.142:6827/1482712612","heartbeat_back_addr":"172.21.15.142:6831/1482712612","heartbeat_front_addr":"172.21.15.142:6829/1482712612","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:08.082015+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:32.634928+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:55.978864+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:21.796863+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:46.862518+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:11.440432+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:37.764777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:03.374958+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:28.714052+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:55.761167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:22.330951+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:47.839334+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/2465242088":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6801/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6800/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/1381997090":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/3507268217":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2570024356":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/2691061938":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2775954801":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6800/2896098005":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6801/2896098005":"2024-04-27T21:25:02.078730+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:32:45.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.281+0000 7f5c8902c640 1 -- 172.21.15.46:0/2748612044 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5c700a3bb0 msgr2=0x7f5c700a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:45.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.281+0000 7f5c8902c640 1 --2- 172.21.15.46:0/2748612044 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5c700a3bb0 0x7f5c700a6070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5c74000c00 tx=0x7f5c740314a0 comp rx=0 tx=0).stop 2024-04-26T21:32:45.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.281+0000 7f5c8902c640 1 -- 172.21.15.46:0/2748612044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c84103420 msgr2=0x7f5c8419ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:45.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.281+0000 7f5c8902c640 1 --2- 172.21.15.46:0/2748612044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c84103420 0x7f5c8419ffe0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c00eb10 tx=0x7f5c7c00ef10 comp rx=0 tx=0).stop 2024-04-26T21:32:45.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.281+0000 7f5c8902c640 1 -- 172.21.15.46:0/2748612044 shutdown_connections 2024-04-26T21:32:45.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.281+0000 7f5c8902c640 1 --2- 172.21.15.46:0/2748612044 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5c700a3bb0 0x7f5c700a6070 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:45.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.281+0000 7f5c8902c640 1 --2- 172.21.15.46:0/2748612044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c84103420 0x7f5c8419ffe0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:45.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.281+0000 7f5c8902c640 1 --2- 172.21.15.46:0/2748612044 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5c84102a60 0x7f5c8419d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:45.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.281+0000 7f5c8902c640 1 --2- 172.21.15.46:0/2748612044 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5c84101870 0x7f5c8406a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:45.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.281+0000 7f5c8902c640 1 -- 172.21.15.46:0/2748612044 >> 172.21.15.46:0/2748612044 conn(0x7f5c840fcde0 msgr2=0x7f5c84105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:45.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.281+0000 7f5c8902c640 1 -- 172.21.15.46:0/2748612044 shutdown_connections 2024-04-26T21:32:45.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.281+0000 7f5c8902c640 1 -- 172.21.15.46:0/2748612044 wait complete. 2024-04-26T21:32:45.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:44 smithi142 ceph-mon[32944]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T21:32:45.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:44 smithi142 ceph-mon[32944]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T21:32:45.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:44 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:45.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:44 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:45.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:44 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:45.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:44 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:32:45.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:44 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:45.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:44 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/1741673035' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T21:32:45.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:44 smithi142 ceph-mon[32944]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T21:32:45.292 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:44 smithi046 ceph-mon[29201]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T21:32:45.292 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:44 smithi046 ceph-mon[29201]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T21:32:45.292 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:44 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:45.292 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:44 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:45.292 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:44 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:45.292 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:44 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:32:45.292 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:44 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:32:45.292 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:44 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1741673035' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T21:32:45.292 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:44 smithi046 ceph-mon[29201]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T21:32:45.293 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:32:45.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.395+0000 7f4b7cc3f640 1 -- 172.21.15.46:0/2901895182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b78102a80 msgr2=0x7f4b78102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:45.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.395+0000 7f4b7cc3f640 1 --2- 172.21.15.46:0/2901895182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b78102a80 0x7f4b78102f00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009900 tx=0x7f4b6002a410 comp rx=0 tx=0).stop 2024-04-26T21:32:45.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.395+0000 7f4b7cc3f640 1 -- 172.21.15.46:0/2901895182 shutdown_connections 2024-04-26T21:32:45.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.395+0000 7f4b7cc3f640 1 --2- 172.21.15.46:0/2901895182 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b78103440 0x7f4b78109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:45.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.395+0000 7f4b7cc3f640 1 --2- 172.21.15.46:0/2901895182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b78102a80 0x7f4b78102f00 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:45.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.395+0000 7f4b7cc3f640 1 --2- 172.21.15.46:0/2901895182 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b78101890 0x7f4b78101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:45.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.395+0000 7f4b7cc3f640 1 -- 172.21.15.46:0/2901895182 >> 172.21.15.46:0/2901895182 conn(0x7f4b780fce20 msgr2=0x7f4b780ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:45.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.396+0000 7f4b7cc3f640 1 -- 172.21.15.46:0/2901895182 shutdown_connections 2024-04-26T21:32:45.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.396+0000 7f4b7cc3f640 1 -- 172.21.15.46:0/2901895182 wait complete. 2024-04-26T21:32:45.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.396+0000 7f4b7cc3f640 1 Processor -- start 2024-04-26T21:32:45.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.396+0000 7f4b7cc3f640 1 -- start start 2024-04-26T21:32:45.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.396+0000 7f4b7cc3f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b78101890 0x7f4b7806bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:45.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.396+0000 7f4b7cc3f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b78102a80 0x7f4b7819d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:45.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.396+0000 7f4b7cc3f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b78103440 0x7f4b7819ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:45.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.396+0000 7f4b7cc3f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7810d3c0 con 0x7f4b78101890 2024-04-26T21:32:45.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.396+0000 7f4b7cc3f640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7810d240 con 0x7f4b78102a80 2024-04-26T21:32:45.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.396+0000 7f4b7cc3f640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7810d540 con 0x7f4b78103440 2024-04-26T21:32:45.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.397+0000 7f4b7659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b78101890 0x7f4b7806bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:45.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.397+0000 7f4b7659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b78101890 0x7f4b7806bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42212/0 (socket says 172.21.15.46:42212) 2024-04-26T21:32:45.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.397+0000 7f4b7659c640 1 -- 172.21.15.46:0/988087173 learned_addr learned my addr 172.21.15.46:0/988087173 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:45.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.397+0000 7f4b76d9d640 1 --2- 172.21.15.46:0/988087173 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b78103440 0x7f4b7819ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:45.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.397+0000 7f4b75d9b640 1 --2- 172.21.15.46:0/988087173 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b78102a80 0x7f4b7819d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:45.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.397+0000 7f4b7659c640 1 -- 172.21.15.46:0/988087173 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b78103440 msgr2=0x7f4b7819ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:45.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.397+0000 7f4b7659c640 1 --2- 172.21.15.46:0/988087173 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b78103440 0x7f4b7819ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:45.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.397+0000 7f4b7659c640 1 -- 172.21.15.46:0/988087173 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b78102a80 msgr2=0x7f4b7819d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:45.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.397+0000 7f4b7659c640 1 --2- 172.21.15.46:0/988087173 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b78102a80 0x7f4b7819d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:45.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.397+0000 7f4b7659c640 1 -- 172.21.15.46:0/988087173 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b781a0820 con 0x7f4b78101890 2024-04-26T21:32:45.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.397+0000 7f4b75d9b640 1 --2- 172.21.15.46:0/988087173 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b78102a80 0x7f4b7819d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:32:45.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.397+0000 7f4b7659c640 1 --2- 172.21.15.46:0/988087173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b78101890 0x7f4b7806bb00 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4b7000d770 tx=0x7f4b7000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:45.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.397+0000 7f4b6f7fe640 1 -- 172.21.15.46:0/988087173 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b70017070 con 0x7f4b78101890 2024-04-26T21:32:45.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.397+0000 7f4b7cc3f640 1 -- 172.21.15.46:0/988087173 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b781a0ab0 con 0x7f4b78101890 2024-04-26T21:32:45.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.397+0000 7f4b6f7fe640 1 -- 172.21.15.46:0/988087173 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b70009c30 con 0x7f4b78101890 2024-04-26T21:32:45.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.397+0000 7f4b6f7fe640 1 -- 172.21.15.46:0/988087173 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b70003cc0 con 0x7f4b78101890 2024-04-26T21:32:45.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.397+0000 7f4b7cc3f640 1 -- 172.21.15.46:0/988087173 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b781a1050 con 0x7f4b78101890 2024-04-26T21:32:45.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.398+0000 7f4b7cc3f640 1 -- 172.21.15.46:0/988087173 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b78108c30 con 0x7f4b78101890 2024-04-26T21:32:45.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.399+0000 7f4b6f7fe640 1 -- 172.21.15.46:0/988087173 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4b70003e60 con 0x7f4b78101890 2024-04-26T21:32:45.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.400+0000 7f4b6f7fe640 1 --2- 172.21.15.46:0/988087173 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4b640a3bb0 0x7f4b640a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:45.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.400+0000 7f4b75d9b640 1 --2- 172.21.15.46:0/988087173 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4b640a3bb0 0x7f4b640a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:45.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.400+0000 7f4b6f7fe640 1 -- 172.21.15.46:0/988087173 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f4b700cfd00 con 0x7f4b78101890 2024-04-26T21:32:45.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.400+0000 7f4b75d9b640 1 --2- 172.21.15.46:0/988087173 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4b640a3bb0 0x7f4b640a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4b7806b870 tx=0x7f4b600314a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:45.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.405+0000 7f4b6f7fe640 1 -- 172.21.15.46:0/988087173 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4b7009c790 con 0x7f4b78101890 2024-04-26T21:32:45.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.591+0000 7f4b7cc3f640 1 -- 172.21.15.46:0/988087173 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4b78063530 con 0x7f4b78101890 2024-04-26T21:32:45.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.593+0000 7f4b6f7fe640 1 -- 172.21.15.46:0/988087173 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22210 (secure 0 0 0) 0x7f4b70009de0 con 0x7f4b78101890 2024-04-26T21:32:45.593 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:32:45.594 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":84,"fsid":"36c7013a-0413-11ef-bc93-c7b262605968","created":"2024-04-26T21:24:30.481156+0000","modified":"2024-04-26T21:32:44.201585+0000","last_up_change":"2024-04-26T21:31:50.100491+0000","last_in_change":"2024-04-26T21:31:29.643428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:27:57.935988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:32:42.195211+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:32:43.162114+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f53c8702-e4da-435f-a9dc-3911d0d36ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6803","nonce":2762665125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6805","nonce":2762665125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6809","nonce":2762665125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6807","nonce":2762665125}]},"public_addr":"172.21.15.46:6803/2762665125","cluster_addr":"172.21.15.46:6805/2762665125","heartbeat_back_addr":"172.21.15.46:6809/2762665125","heartbeat_front_addr":"172.21.15.46:6807/2762665125","state":["exists","up"]},{"osd":1,"uuid":"fc4d6f52-fc8a-4efd-80c9-c092b57e7f42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6801","nonce":1628008970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6803","nonce":1628008970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6807","nonce":1628008970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6805","nonce":1628008970}]},"public_addr":"172.21.15.50:6801/1628008970","cluster_addr":"172.21.15.50:6803/1628008970","heartbeat_back_addr":"172.21.15.50:6807/1628008970","heartbeat_front_addr":"172.21.15.50:6805/1628008970","state":["exists","up"]},{"osd":2,"uuid":"26d1baa1-4ea1-4b7e-9fcf-4f8bbf7329c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6801","nonce":1513279137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6803","nonce":1513279137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6807","nonce":1513279137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6805","nonce":1513279137}]},"public_addr":"172.21.15.142:6801/1513279137","cluster_addr":"172.21.15.142:6803/1513279137","heartbeat_back_addr":"172.21.15.142:6807/1513279137","heartbeat_front_addr":"172.21.15.142:6805/1513279137","state":["exists","up"]},{"osd":3,"uuid":"34d2401f-5fcc-491c-8fcf-c3c492611f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6811","nonce":2435181760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6813","nonce":2435181760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6817","nonce":2435181760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6815","nonce":2435181760}]},"public_addr":"172.21.15.46:6811/2435181760","cluster_addr":"172.21.15.46:6813/2435181760","heartbeat_back_addr":"172.21.15.46:6817/2435181760","heartbeat_front_addr":"172.21.15.46:6815/2435181760","state":["exists","up"]},{"osd":4,"uuid":"986bb709-e4cf-400d-982e-c3f5ac413e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6809","nonce":3177333250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6811","nonce":3177333250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6815","nonce":3177333250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6813","nonce":3177333250}]},"public_addr":"172.21.15.50:6809/3177333250","cluster_addr":"172.21.15.50:6811/3177333250","heartbeat_back_addr":"172.21.15.50:6815/3177333250","heartbeat_front_addr":"172.21.15.50:6813/3177333250","state":["exists","up"]},{"osd":5,"uuid":"b7d65851-ebe1-45e2-833d-8022446f2698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6809","nonce":4113107737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6811","nonce":4113107737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6815","nonce":4113107737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6813","nonce":4113107737}]},"public_addr":"172.21.15.142:6809/4113107737","cluster_addr":"172.21.15.142:6811/4113107737","heartbeat_back_addr":"172.21.15.142:6815/4113107737","heartbeat_front_addr":"172.21.15.142:6813/4113107737","state":["exists","up"]},{"osd":6,"uuid":"2a3e96d7-d5bf-44ff-9134-29f1420c0e4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6819","nonce":2325162882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6821","nonce":2325162882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6825","nonce":2325162882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6823","nonce":2325162882}]},"public_addr":"172.21.15.46:6819/2325162882","cluster_addr":"172.21.15.46:6821/2325162882","heartbeat_back_addr":"172.21.15.46:6825/2325162882","heartbeat_front_addr":"172.21.15.46:6823/2325162882","state":["exists","up"]},{"osd":7,"uuid":"e53ae885-23b6-46fd-b4b1-c62f31a6921f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6817","nonce":820310228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6819","nonce":820310228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6823","nonce":820310228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6821","nonce":820310228}]},"public_addr":"172.21.15.50:6817/820310228","cluster_addr":"172.21.15.50:6819/820310228","heartbeat_back_addr":"172.21.15.50:6823/820310228","heartbeat_front_addr":"172.21.15.50:6821/820310228","state":["exists","up"]},{"osd":8,"uuid":"356e2692-16a9-4045-b58f-c8f9cb5d7241","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6817","nonce":1824918711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6819","nonce":1824918711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6823","nonce":1824918711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6821","nonce":1824918711}]},"public_addr":"172.21.15.142:6817/1824918711","cluster_addr":"172.21.15.142:6819/1824918711","heartbeat_back_addr":"172.21.15.142:6823/1824918711","heartbeat_front_addr":"172.21.15.142:6821/1824918711","state":["exists","up"]},{"osd":9,"uuid":"8e55570e-77e5-4892-8b9f-77dcd03a4eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6827","nonce":3601947736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6829","nonce":3601947736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6833","nonce":3601947736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6831","nonce":3601947736}]},"public_addr":"172.21.15.46:6827/3601947736","cluster_addr":"172.21.15.46:6829/3601947736","heartbeat_back_addr":"172.21.15.46:6833/3601947736","heartbeat_front_addr":"172.21.15.46:6831/3601947736","state":["exists","up"]},{"osd":10,"uuid":"366389fa-fb9a-408b-b5c2-343c054c5a64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6825","nonce":3135765006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6827","nonce":3135765006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6831","nonce":3135765006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6829","nonce":3135765006}]},"public_addr":"172.21.15.50:6825/3135765006","cluster_addr":"172.21.15.50:6827/3135765006","heartbeat_back_addr":"172.21.15.50:6831/3135765006","heartbeat_front_addr":"172.21.15.50:6829/3135765006","state":["exists","up"]},{"osd":11,"uuid":"7b6e3d93-59e1-4853-b4a2-261b728c47a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6825","nonce":1482712612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6827","nonce":1482712612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6831","nonce":1482712612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6829","nonce":1482712612}]},"public_addr":"172.21.15.142:6825/1482712612","cluster_addr":"172.21.15.142:6827/1482712612","heartbeat_back_addr":"172.21.15.142:6831/1482712612","heartbeat_front_addr":"172.21.15.142:6829/1482712612","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:08.082015+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:32.634928+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:55.978864+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:21.796863+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:46.862518+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:11.440432+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:37.764777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:03.374958+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:28.714052+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:55.761167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:22.330951+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:47.839334+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/2465242088":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6801/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6800/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/1381997090":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/3507268217":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2570024356":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/2691061938":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2775954801":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6800/2896098005":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6801/2896098005":"2024-04-27T21:25:02.078730+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:32:45.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.595+0000 7f4b7cc3f640 1 -- 172.21.15.46:0/988087173 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4b640a3bb0 msgr2=0x7f4b640a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:45.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.595+0000 7f4b7cc3f640 1 --2- 172.21.15.46:0/988087173 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4b640a3bb0 0x7f4b640a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4b7806b870 tx=0x7f4b600314a0 comp rx=0 tx=0).stop 2024-04-26T21:32:45.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.595+0000 7f4b7cc3f640 1 -- 172.21.15.46:0/988087173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b78101890 msgr2=0x7f4b7806bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:45.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.595+0000 7f4b7cc3f640 1 --2- 172.21.15.46:0/988087173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b78101890 0x7f4b7806bb00 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4b7000d770 tx=0x7f4b7000dc30 comp rx=0 tx=0).stop 2024-04-26T21:32:45.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.595+0000 7f4b7cc3f640 1 -- 172.21.15.46:0/988087173 shutdown_connections 2024-04-26T21:32:45.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.595+0000 7f4b7cc3f640 1 --2- 172.21.15.46:0/988087173 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4b640a3bb0 0x7f4b640a6070 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:45.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.595+0000 7f4b7cc3f640 1 --2- 172.21.15.46:0/988087173 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4b78103440 0x7f4b7819ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:45.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.595+0000 7f4b7cc3f640 1 --2- 172.21.15.46:0/988087173 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b78102a80 0x7f4b7819d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:45.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.595+0000 7f4b7cc3f640 1 --2- 172.21.15.46:0/988087173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b78101890 0x7f4b7806bb00 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:45.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.595+0000 7f4b7cc3f640 1 -- 172.21.15.46:0/988087173 >> 172.21.15.46:0/988087173 conn(0x7f4b780fce20 msgr2=0x7f4b78105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:45.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.595+0000 7f4b7cc3f640 1 -- 172.21.15.46:0/988087173 shutdown_connections 2024-04-26T21:32:45.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.595+0000 7f4b7cc3f640 1 -- 172.21.15.46:0/988087173 wait complete. 2024-04-26T21:32:45.607 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-26T21:32:45.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.711+0000 7f1700ea0640 1 -- 172.21.15.46:0/3924312559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16fc102990 msgr2=0x7f16fc102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:45.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.711+0000 7f1700ea0640 1 --2- 172.21.15.46:0/3924312559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16fc102990 0x7f16fc102db0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f16ec009900 tx=0x7f16ec02a410 comp rx=0 tx=0).stop 2024-04-26T21:32:45.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.711+0000 7f1700ea0640 1 -- 172.21.15.46:0/3924312559 shutdown_connections 2024-04-26T21:32:45.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.711+0000 7f1700ea0640 1 --2- 172.21.15.46:0/3924312559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16fc104540 0x7f16fc10add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:45.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.711+0000 7f1700ea0640 1 --2- 172.21.15.46:0/3924312559 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16fc103b80 0x7f16fc104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:45.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.711+0000 7f1700ea0640 1 --2- 172.21.15.46:0/3924312559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16fc102990 0x7f16fc102db0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:45.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.711+0000 7f1700ea0640 1 -- 172.21.15.46:0/3924312559 >> 172.21.15.46:0/3924312559 conn(0x7f16fc0fdf00 msgr2=0x7f16fc100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:45.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.711+0000 7f1700ea0640 1 -- 172.21.15.46:0/3924312559 shutdown_connections 2024-04-26T21:32:45.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.711+0000 7f1700ea0640 1 -- 172.21.15.46:0/3924312559 wait complete. 2024-04-26T21:32:45.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.712+0000 7f1700ea0640 1 Processor -- start 2024-04-26T21:32:45.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.712+0000 7f1700ea0640 1 -- start start 2024-04-26T21:32:45.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.712+0000 7f1700ea0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16fc102990 0x7f16fc198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:45.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.712+0000 7f1700ea0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16fc103b80 0x7f16fc199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:45.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.712+0000 7f1700ea0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16fc104540 0x7f16fc1a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:45.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.712+0000 7f1700ea0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f16fc10e4b0 con 0x7f16fc104540 2024-04-26T21:32:45.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.712+0000 7f1700ea0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f16fc10e330 con 0x7f16fc103b80 2024-04-26T21:32:45.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.712+0000 7f1700ea0640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f16fc10e630 con 0x7f16fc102990 2024-04-26T21:32:45.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.712+0000 7f16fad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16fc104540 0x7f16fc1a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:45.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.712+0000 7f16fad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16fc104540 0x7f16fc1a01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49342/0 (socket says 172.21.15.46:49342) 2024-04-26T21:32:45.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.712+0000 7f16fad9d640 1 -- 172.21.15.46:0/1736888585 learned_addr learned my addr 172.21.15.46:0/1736888585 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:45.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.712+0000 7f16fa59c640 1 --2- 172.21.15.46:0/1736888585 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16fc102990 0x7f16fc198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:45.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.712+0000 7f16f9d9b640 1 --2- 172.21.15.46:0/1736888585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16fc103b80 0x7f16fc199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:45.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.712+0000 7f16fad9d640 1 -- 172.21.15.46:0/1736888585 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16fc102990 msgr2=0x7f16fc198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:45.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.712+0000 7f16fad9d640 1 --2- 172.21.15.46:0/1736888585 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16fc102990 0x7f16fc198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:45.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.712+0000 7f16fad9d640 1 -- 172.21.15.46:0/1736888585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16fc103b80 msgr2=0x7f16fc199170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:45.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.712+0000 7f16fad9d640 1 --2- 172.21.15.46:0/1736888585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16fc103b80 0x7f16fc199170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:45.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.712+0000 7f16fad9d640 1 -- 172.21.15.46:0/1736888585 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16fc1a0840 con 0x7f16fc104540 2024-04-26T21:32:45.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.712+0000 7f16f9d9b640 1 --2- 172.21.15.46:0/1736888585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16fc103b80 0x7f16fc199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:32:45.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.713+0000 7f16fad9d640 1 --2- 172.21.15.46:0/1736888585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16fc104540 0x7f16fc1a01a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f16f400eaa0 tx=0x7f16f400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:45.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.713+0000 7f16eb7fe640 1 -- 172.21.15.46:0/1736888585 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f400cdb0 con 0x7f16fc104540 2024-04-26T21:32:45.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.713+0000 7f16eb7fe640 1 -- 172.21.15.46:0/1736888585 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16f4005020 con 0x7f16fc104540 2024-04-26T21:32:45.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.713+0000 7f1700ea0640 1 -- 172.21.15.46:0/1736888585 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16fc1a0b30 con 0x7f16fc104540 2024-04-26T21:32:45.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.713+0000 7f16eb7fe640 1 -- 172.21.15.46:0/1736888585 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f40116d0 con 0x7f16fc104540 2024-04-26T21:32:45.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.713+0000 7f1700ea0640 1 -- 172.21.15.46:0/1736888585 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16fc1a1030 con 0x7f16fc104540 2024-04-26T21:32:45.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.714+0000 7f1700ea0640 1 -- 172.21.15.46:0/1736888585 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16fc109d50 con 0x7f16fc104540 2024-04-26T21:32:45.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.715+0000 7f16eb7fe640 1 -- 172.21.15.46:0/1736888585 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f16f40051d0 con 0x7f16fc104540 2024-04-26T21:32:45.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.715+0000 7f16eb7fe640 1 --2- 172.21.15.46:0/1736888585 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f16e40a3bb0 0x7f16e40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:45.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.716+0000 7f16fa59c640 1 --2- 172.21.15.46:0/1736888585 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f16e40a3bb0 0x7f16e40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:45.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.716+0000 7f16eb7fe640 1 -- 172.21.15.46:0/1736888585 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f16f40d0540 con 0x7f16fc104540 2024-04-26T21:32:45.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.716+0000 7f16fa59c640 1 --2- 172.21.15.46:0/1736888585 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f16e40a3bb0 0x7f16e40a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f16fc19a100 tx=0x7f16ec0314a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:45.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.721+0000 7f16eb7fe640 1 -- 172.21.15.46:0/1736888585 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f16f4016030 con 0x7f16fc104540 2024-04-26T21:32:45.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:45.953+0000 7f1700ea0640 1 -- 172.21.15.46:0/1736888585 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f16fc02dad0 con 0x7f16fc104540 2024-04-26T21:32:46.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:45 smithi046 ceph-mon[29201]: pgmap v290: 97 pgs: 2 creating+peering, 1 active+clean, 94 unknown; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:46.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:45 smithi046 ceph-mon[29201]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T21:32:46.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:45 smithi046 ceph-mon[29201]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T21:32:46.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:45 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1741673035' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T21:32:46.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:45 smithi046 ceph-mon[29201]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T21:32:46.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:45 smithi046 ceph-mon[29201]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T21:32:46.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:45 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2748612044' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:32:46.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:45 smithi046 ceph-mon[29201]: mds.? [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] up:active 2024-04-26T21:32:46.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:45 smithi046 ceph-mon[29201]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T21:32:46.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:45 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/988087173' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:32:46.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:45 smithi050 ceph-mon[32753]: pgmap v290: 97 pgs: 2 creating+peering, 1 active+clean, 94 unknown; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:46.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:45 smithi050 ceph-mon[32753]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T21:32:46.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:45 smithi050 ceph-mon[32753]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T21:32:46.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:45 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/1741673035' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T21:32:46.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:45 smithi050 ceph-mon[32753]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T21:32:46.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:45 smithi050 ceph-mon[32753]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T21:32:46.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:45 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/2748612044' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:32:46.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:45 smithi050 ceph-mon[32753]: mds.? [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] up:active 2024-04-26T21:32:46.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:45 smithi050 ceph-mon[32753]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T21:32:46.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:45 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/988087173' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:32:46.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:45 smithi142 ceph-mon[32944]: pgmap v290: 97 pgs: 2 creating+peering, 1 active+clean, 94 unknown; 577 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:32:46.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:45 smithi142 ceph-mon[32944]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T21:32:46.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:45 smithi142 ceph-mon[32944]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T21:32:46.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:45 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/1741673035' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T21:32:46.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:45 smithi142 ceph-mon[32944]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T21:32:46.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:45 smithi142 ceph-mon[32944]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T21:32:46.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:45 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/2748612044' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:32:46.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:45 smithi142 ceph-mon[32944]: mds.? [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] up:active 2024-04-26T21:32:46.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:45 smithi142 ceph-mon[32944]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T21:32:46.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:45 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/988087173' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:32:46.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.859+0000 7f16eb7fe640 1 -- 172.21.15.46:0/1736888585 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) ==== 118+0+0 (secure 0 0 0) 0x7f16f409d070 con 0x7f16fc104540 2024-04-26T21:32:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.862+0000 7f1700ea0640 1 -- 172.21.15.46:0/1736888585 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f16e40a3bb0 msgr2=0x7f16e40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.862+0000 7f1700ea0640 1 --2- 172.21.15.46:0/1736888585 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f16e40a3bb0 0x7f16e40a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f16fc19a100 tx=0x7f16ec0314a0 comp rx=0 tx=0).stop 2024-04-26T21:32:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.862+0000 7f1700ea0640 1 -- 172.21.15.46:0/1736888585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16fc104540 msgr2=0x7f16fc1a01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.862+0000 7f1700ea0640 1 --2- 172.21.15.46:0/1736888585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16fc104540 0x7f16fc1a01a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f16f400eaa0 tx=0x7f16f400ef60 comp rx=0 tx=0).stop 2024-04-26T21:32:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.862+0000 7f1700ea0640 1 -- 172.21.15.46:0/1736888585 shutdown_connections 2024-04-26T21:32:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.862+0000 7f1700ea0640 1 --2- 172.21.15.46:0/1736888585 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f16e40a3bb0 0x7f16e40a6070 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.862+0000 7f1700ea0640 1 --2- 172.21.15.46:0/1736888585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16fc104540 0x7f16fc1a01a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.862+0000 7f1700ea0640 1 --2- 172.21.15.46:0/1736888585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f16fc103b80 0x7f16fc199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.862+0000 7f1700ea0640 1 --2- 172.21.15.46:0/1736888585 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f16fc102990 0x7f16fc198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.862+0000 7f1700ea0640 1 -- 172.21.15.46:0/1736888585 >> 172.21.15.46:0/1736888585 conn(0x7f16fc0fdf00 msgr2=0x7f16fc106f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.862+0000 7f1700ea0640 1 -- 172.21.15.46:0/1736888585 shutdown_connections 2024-04-26T21:32:46.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.862+0000 7f1700ea0640 1 -- 172.21.15.46:0/1736888585 wait complete. 2024-04-26T21:32:46.876 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-04-26T21:32:46.876 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-26T21:32:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.987+0000 7ff0ca17f640 1 -- 172.21.15.46:0/106884239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0c4105d20 msgr2=0x7ff0c4108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.987+0000 7ff0ca17f640 1 --2- 172.21.15.46:0/106884239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0c4105d20 0x7ff0c4108110 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff0b4009900 tx=0x7ff0b402a410 comp rx=0 tx=0).stop 2024-04-26T21:32:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.987+0000 7ff0ca17f640 1 -- 172.21.15.46:0/106884239 shutdown_connections 2024-04-26T21:32:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.987+0000 7ff0ca17f640 1 --2- 172.21.15.46:0/106884239 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff0c4108650 0x7ff0c410aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.987+0000 7ff0ca17f640 1 --2- 172.21.15.46:0/106884239 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0c4105d20 0x7ff0c4108110 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.987+0000 7ff0ca17f640 1 --2- 172.21.15.46:0/106884239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff0c41033f0 0x7ff0c41057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.987+0000 7ff0ca17f640 1 -- 172.21.15.46:0/106884239 >> 172.21.15.46:0/106884239 conn(0x7ff0c40fcde0 msgr2=0x7ff0c40ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.987+0000 7ff0ca17f640 1 -- 172.21.15.46:0/106884239 shutdown_connections 2024-04-26T21:32:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.988+0000 7ff0ca17f640 1 -- 172.21.15.46:0/106884239 wait complete. 2024-04-26T21:32:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.988+0000 7ff0ca17f640 1 Processor -- start 2024-04-26T21:32:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.988+0000 7ff0ca17f640 1 -- start start 2024-04-26T21:32:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.988+0000 7ff0ca17f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0c41033f0 0x7ff0c4198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.988+0000 7ff0ca17f640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff0c4105d20 0x7ff0c4199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.988+0000 7ff0ca17f640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff0c4108650 0x7ff0c41a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.988+0000 7ff0ca17f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c410e4e0 con 0x7ff0c41033f0 2024-04-26T21:32:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.988+0000 7ff0ca17f640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c410e360 con 0x7ff0c4105d20 2024-04-26T21:32:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.988+0000 7ff0ca17f640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c410e660 con 0x7ff0c4108650 2024-04-26T21:32:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.988+0000 7ff0c37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0c41033f0 0x7ff0c4198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.989+0000 7ff0c37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0c41033f0 0x7ff0c4198be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49362/0 (socket says 172.21.15.46:49362) 2024-04-26T21:32:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.989+0000 7ff0c37fe640 1 -- 172.21.15.46:0/2300160823 learned_addr learned my addr 172.21.15.46:0/2300160823 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.989+0000 7ff0c2ffd640 1 --2- 172.21.15.46:0/2300160823 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff0c4105d20 0x7ff0c4199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.989+0000 7ff0c3fff640 1 --2- 172.21.15.46:0/2300160823 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff0c4108650 0x7ff0c41a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.989+0000 7ff0c37fe640 1 -- 172.21.15.46:0/2300160823 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff0c4108650 msgr2=0x7ff0c41a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.989+0000 7ff0c37fe640 1 --2- 172.21.15.46:0/2300160823 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff0c4108650 0x7ff0c41a0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.989+0000 7ff0c37fe640 1 -- 172.21.15.46:0/2300160823 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff0c4105d20 msgr2=0x7ff0c4199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.989+0000 7ff0c37fe640 1 --2- 172.21.15.46:0/2300160823 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff0c4105d20 0x7ff0c4199120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.989+0000 7ff0c37fe640 1 -- 172.21.15.46:0/2300160823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0c41a0800 con 0x7ff0c41033f0 2024-04-26T21:32:46.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.989+0000 7ff0c2ffd640 1 --2- 172.21.15.46:0/2300160823 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff0c4105d20 0x7ff0c4199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:32:46.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.989+0000 7ff0c37fe640 1 --2- 172.21.15.46:0/2300160823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0c41033f0 0x7ff0c4198be0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac00d950 tx=0x7ff0ac00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:46.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.989+0000 7ff0c0ff9640 1 -- 172.21.15.46:0/2300160823 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac017070 con 0x7ff0c41033f0 2024-04-26T21:32:46.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.989+0000 7ff0ca17f640 1 -- 172.21.15.46:0/2300160823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0c41a0af0 con 0x7ff0c41033f0 2024-04-26T21:32:46.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.989+0000 7ff0c0ff9640 1 -- 172.21.15.46:0/2300160823 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0ac009c30 con 0x7ff0c41033f0 2024-04-26T21:32:46.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.989+0000 7ff0c0ff9640 1 -- 172.21.15.46:0/2300160823 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac00f620 con 0x7ff0c41033f0 2024-04-26T21:32:46.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.989+0000 7ff0ca17f640 1 -- 172.21.15.46:0/2300160823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0c41a1090 con 0x7ff0c41033f0 2024-04-26T21:32:46.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.990+0000 7ff0ca17f640 1 -- 172.21.15.46:0/2300160823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0c40fe8d0 con 0x7ff0c41033f0 2024-04-26T21:32:46.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.992+0000 7ff0c0ff9640 1 -- 172.21.15.46:0/2300160823 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff0ac010460 con 0x7ff0c41033f0 2024-04-26T21:32:46.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.992+0000 7ff0c0ff9640 1 --2- 172.21.15.46:0/2300160823 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ff0b00a3bb0 0x7ff0b00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:46.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.992+0000 7ff0c0ff9640 1 -- 172.21.15.46:0/2300160823 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7ff0ac0cff90 con 0x7ff0c41033f0 2024-04-26T21:32:46.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.992+0000 7ff0c2ffd640 1 --2- 172.21.15.46:0/2300160823 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ff0b00a3bb0 0x7ff0b00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:46.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.993+0000 7ff0c2ffd640 1 --2- 172.21.15.46:0/2300160823 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ff0b00a3bb0 0x7ff0b00a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff0b402a930 tx=0x7ff0b4005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:46.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:46.997+0000 7ff0c0ff9640 1 -- 172.21.15.46:0/2300160823 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff0ac09cac0 con 0x7ff0c41033f0 2024-04-26T21:32:47.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:46 smithi050 ceph-mon[32753]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T21:32:47.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:46 smithi050 ceph-mon[32753]: pgmap v293: 97 pgs: 19 creating+peering, 58 active+clean, 20 unknown; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 4 op/s 2024-04-26T21:32:47.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:46 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/1736888585' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T21:32:47.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:47.222+0000 7ff0ca17f640 1 -- 172.21.15.46:0/2300160823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7ff0c402dd40 con 0x7ff0c41033f0 2024-04-26T21:32:47.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:46 smithi142 ceph-mon[32944]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T21:32:47.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:46 smithi142 ceph-mon[32944]: pgmap v293: 97 pgs: 19 creating+peering, 58 active+clean, 20 unknown; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 4 op/s 2024-04-26T21:32:47.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:46 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/1736888585' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T21:32:47.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:46 smithi046 ceph-mon[29201]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T21:32:47.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:46 smithi046 ceph-mon[29201]: pgmap v293: 97 pgs: 19 creating+peering, 58 active+clean, 20 unknown; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 4 op/s 2024-04-26T21:32:47.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:46 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1736888585' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T21:32:47.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:47.868+0000 7ff0c0ff9640 1 -- 172.21.15.46:0/2300160823 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v17) ==== 105+0+0 (secure 0 0 0) 0x7ff0ac009de0 con 0x7ff0c41033f0 2024-04-26T21:32:47.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:47.871+0000 7ff0ca17f640 1 -- 172.21.15.46:0/2300160823 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ff0b00a3bb0 msgr2=0x7ff0b00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:47.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:47.871+0000 7ff0ca17f640 1 --2- 172.21.15.46:0/2300160823 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ff0b00a3bb0 0x7ff0b00a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff0b402a930 tx=0x7ff0b4005fd0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:47.871+0000 7ff0ca17f640 1 -- 172.21.15.46:0/2300160823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0c41033f0 msgr2=0x7ff0c4198be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:47.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:47.871+0000 7ff0ca17f640 1 --2- 172.21.15.46:0/2300160823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0c41033f0 0x7ff0c4198be0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac00d950 tx=0x7ff0ac00de10 comp rx=0 tx=0).stop 2024-04-26T21:32:47.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:47.871+0000 7ff0ca17f640 1 -- 172.21.15.46:0/2300160823 shutdown_connections 2024-04-26T21:32:47.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:47.871+0000 7ff0ca17f640 1 --2- 172.21.15.46:0/2300160823 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ff0b00a3bb0 0x7ff0b00a6070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:47.871+0000 7ff0ca17f640 1 --2- 172.21.15.46:0/2300160823 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff0c4108650 0x7ff0c41a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:47.871+0000 7ff0ca17f640 1 --2- 172.21.15.46:0/2300160823 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff0c4105d20 0x7ff0c4199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:47.871+0000 7ff0ca17f640 1 --2- 172.21.15.46:0/2300160823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0c41033f0 0x7ff0c4198be0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:47.871+0000 7ff0ca17f640 1 -- 172.21.15.46:0/2300160823 >> 172.21.15.46:0/2300160823 conn(0x7ff0c40fcde0 msgr2=0x7ff0c4103c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:47.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:47.871+0000 7ff0ca17f640 1 -- 172.21.15.46:0/2300160823 shutdown_connections 2024-04-26T21:32:47.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:47.871+0000 7ff0ca17f640 1 -- 172.21.15.46:0/2300160823 wait complete. 2024-04-26T21:32:47.889 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-26T21:32:48.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.000+0000 7f068e2bd640 1 -- 172.21.15.46:0/1627189627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0688103b80 msgr2=0x7f0688104000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:48.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.000+0000 7f068e2bd640 1 --2- 172.21.15.46:0/1627189627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0688103b80 0x7f0688104000 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f067c009900 tx=0x7f067c02a410 comp rx=0 tx=0).stop 2024-04-26T21:32:48.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.000+0000 7f068e2bd640 1 -- 172.21.15.46:0/1627189627 shutdown_connections 2024-04-26T21:32:48.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.000+0000 7f068e2bd640 1 --2- 172.21.15.46:0/1627189627 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0688104540 0x7f068810add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:48.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.000+0000 7f068e2bd640 1 --2- 172.21.15.46:0/1627189627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0688103b80 0x7f0688104000 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:48.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.000+0000 7f068e2bd640 1 --2- 172.21.15.46:0/1627189627 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0688102990 0x7f0688102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:48.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.000+0000 7f068e2bd640 1 -- 172.21.15.46:0/1627189627 >> 172.21.15.46:0/1627189627 conn(0x7f06880fdf00 msgr2=0x7f0688100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:48.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.000+0000 7f068e2bd640 1 -- 172.21.15.46:0/1627189627 shutdown_connections 2024-04-26T21:32:48.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.000+0000 7f068e2bd640 1 -- 172.21.15.46:0/1627189627 wait complete. 2024-04-26T21:32:48.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.001+0000 7f068e2bd640 1 Processor -- start 2024-04-26T21:32:48.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.001+0000 7f068e2bd640 1 -- start start 2024-04-26T21:32:48.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.001+0000 7f068e2bd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0688102990 0x7f0688198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:48.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.001+0000 7f068e2bd640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0688103b80 0x7f0688199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:48.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.001+0000 7f068e2bd640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0688104540 0x7f06881a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:48.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.001+0000 7f068e2bd640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f068810e520 con 0x7f0688102990 2024-04-26T21:32:48.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.001+0000 7f068e2bd640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f068810e3a0 con 0x7f0688103b80 2024-04-26T21:32:48.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.001+0000 7f068e2bd640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f068810e6a0 con 0x7f0688104540 2024-04-26T21:32:48.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.001+0000 7f0687fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0688102990 0x7f0688198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:48.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.001+0000 7f0687fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0688102990 0x7f0688198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49374/0 (socket says 172.21.15.46:49374) 2024-04-26T21:32:48.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.001+0000 7f0687fff640 1 -- 172.21.15.46:0/2417904786 learned_addr learned my addr 172.21.15.46:0/2417904786 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:48.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.001+0000 7f06877fe640 1 --2- 172.21.15.46:0/2417904786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0688103b80 0x7f0688199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:48.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.001+0000 7f068c85a640 1 --2- 172.21.15.46:0/2417904786 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0688104540 0x7f06881a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:48.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.001+0000 7f0687fff640 1 -- 172.21.15.46:0/2417904786 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0688104540 msgr2=0x7f06881a01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:48.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.001+0000 7f0687fff640 1 --2- 172.21.15.46:0/2417904786 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0688104540 0x7f06881a01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:48.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.001+0000 7f0687fff640 1 -- 172.21.15.46:0/2417904786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0688103b80 msgr2=0x7f0688199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:48.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.001+0000 7f0687fff640 1 --2- 172.21.15.46:0/2417904786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0688103b80 0x7f0688199180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:48.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.001+0000 7f0687fff640 1 -- 172.21.15.46:0/2417904786 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06881a0850 con 0x7f0688102990 2024-04-26T21:32:48.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.002+0000 7f0687fff640 1 --2- 172.21.15.46:0/2417904786 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0688102990 0x7f0688198c40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f067800d770 tx=0x7f067800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:48.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.002+0000 7f06857fa640 1 -- 172.21.15.46:0/2417904786 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0678017070 con 0x7f0688102990 2024-04-26T21:32:48.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.002+0000 7f06857fa640 1 -- 172.21.15.46:0/2417904786 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0678009c30 con 0x7f0688102990 2024-04-26T21:32:48.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.002+0000 7f068e2bd640 1 -- 172.21.15.46:0/2417904786 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06881a0b40 con 0x7f0688102990 2024-04-26T21:32:48.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.002+0000 7f06857fa640 1 -- 172.21.15.46:0/2417904786 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0678003cc0 con 0x7f0688102990 2024-04-26T21:32:48.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.002+0000 7f068e2bd640 1 -- 172.21.15.46:0/2417904786 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06881a1090 con 0x7f0688102990 2024-04-26T21:32:48.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.003+0000 7f068e2bd640 1 -- 172.21.15.46:0/2417904786 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0688109d20 con 0x7f0688102990 2024-04-26T21:32:48.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.004+0000 7f06857fa640 1 -- 172.21.15.46:0/2417904786 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0678003e60 con 0x7f0688102990 2024-04-26T21:32:48.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.005+0000 7f06857fa640 1 --2- 172.21.15.46:0/2417904786 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f06700a3bb0 0x7f06700a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:48.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.005+0000 7f06877fe640 1 --2- 172.21.15.46:0/2417904786 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f06700a3bb0 0x7f06700a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:48.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.005+0000 7f06857fa640 1 -- 172.21.15.46:0/2417904786 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f06780cfcf0 con 0x7f0688102990 2024-04-26T21:32:48.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.005+0000 7f06877fe640 1 --2- 172.21.15.46:0/2417904786 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f06700a3bb0 0x7f06700a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f067c000c00 tx=0x7f067c0314a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:48.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.010+0000 7f06857fa640 1 -- 172.21.15.46:0/2417904786 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f06780d4070 con 0x7f0688102990 2024-04-26T21:32:48.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:47 smithi050 ceph-mon[32753]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T21:32:48.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:47 smithi050 ceph-mon[32753]: Cluster is now healthy 2024-04-26T21:32:48.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:47 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/1736888585' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T21:32:48.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:47 smithi050 ceph-mon[32753]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T21:32:48.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:47 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/2300160823' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T21:32:48.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.236+0000 7f068e2bd640 1 -- 172.21.15.46:0/2417904786 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f068802dad0 con 0x7f0688102990 2024-04-26T21:32:48.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:47 smithi142 ceph-mon[32944]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T21:32:48.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:47 smithi142 ceph-mon[32944]: Cluster is now healthy 2024-04-26T21:32:48.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:47 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/1736888585' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T21:32:48.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:47 smithi142 ceph-mon[32944]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T21:32:48.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:47 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/2300160823' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T21:32:48.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:47 smithi046 ceph-mon[29201]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-26T21:32:48.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:47 smithi046 ceph-mon[29201]: Cluster is now healthy 2024-04-26T21:32:48.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:47 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1736888585' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T21:32:48.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:47 smithi046 ceph-mon[29201]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T21:32:48.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:47 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2300160823' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T21:32:48.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.878+0000 7f06857fa640 1 -- 172.21.15.46:0/2417904786 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v19) ==== 121+0+0 (secure 0 0 0) 0x7f067809c820 con 0x7f0688102990 2024-04-26T21:32:48.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.881+0000 7f068e2bd640 1 -- 172.21.15.46:0/2417904786 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f06700a3bb0 msgr2=0x7f06700a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:48.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.881+0000 7f068e2bd640 1 --2- 172.21.15.46:0/2417904786 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f06700a3bb0 0x7f06700a6070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f067c000c00 tx=0x7f067c0314a0 comp rx=0 tx=0).stop 2024-04-26T21:32:48.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.881+0000 7f068e2bd640 1 -- 172.21.15.46:0/2417904786 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0688102990 msgr2=0x7f0688198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:48.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.881+0000 7f068e2bd640 1 --2- 172.21.15.46:0/2417904786 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0688102990 0x7f0688198c40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f067800d770 tx=0x7f067800dc30 comp rx=0 tx=0).stop 2024-04-26T21:32:48.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.881+0000 7f068e2bd640 1 -- 172.21.15.46:0/2417904786 shutdown_connections 2024-04-26T21:32:48.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.881+0000 7f068e2bd640 1 --2- 172.21.15.46:0/2417904786 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f06700a3bb0 0x7f06700a6070 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:48.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.881+0000 7f068e2bd640 1 --2- 172.21.15.46:0/2417904786 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0688104540 0x7f06881a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:48.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.881+0000 7f068e2bd640 1 --2- 172.21.15.46:0/2417904786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0688103b80 0x7f0688199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:48.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.881+0000 7f068e2bd640 1 --2- 172.21.15.46:0/2417904786 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0688102990 0x7f0688198c40 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:48.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.881+0000 7f068e2bd640 1 -- 172.21.15.46:0/2417904786 >> 172.21.15.46:0/2417904786 conn(0x7f06880fdf00 msgr2=0x7f0688106ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:48.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.881+0000 7f068e2bd640 1 -- 172.21.15.46:0/2417904786 shutdown_connections 2024-04-26T21:32:48.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:48.881+0000 7f068e2bd640 1 -- 172.21.15.46:0/2417904786 wait complete. 2024-04-26T21:32:48.898 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-26T21:32:49.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.010+0000 7f2277c47640 1 -- 172.21.15.46:0/622792046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2270105da0 msgr2=0x7f2270108190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:49.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.010+0000 7f2277c47640 1 --2- 172.21.15.46:0/622792046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2270105da0 0x7f2270108190 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2260009900 tx=0x7f226002a410 comp rx=0 tx=0).stop 2024-04-26T21:32:49.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.010+0000 7f2277c47640 1 -- 172.21.15.46:0/622792046 shutdown_connections 2024-04-26T21:32:49.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.010+0000 7f2277c47640 1 --2- 172.21.15.46:0/622792046 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f22701086d0 0x7f227010aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:49.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.010+0000 7f2277c47640 1 --2- 172.21.15.46:0/622792046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2270105da0 0x7f2270108190 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:49.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.010+0000 7f2277c47640 1 --2- 172.21.15.46:0/622792046 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2270103470 0x7f2270105860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:49.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.010+0000 7f2277c47640 1 -- 172.21.15.46:0/622792046 >> 172.21.15.46:0/622792046 conn(0x7f22700fce20 msgr2=0x7f22700ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:49.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.010+0000 7f2277c47640 1 -- 172.21.15.46:0/622792046 shutdown_connections 2024-04-26T21:32:49.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.010+0000 7f2277c47640 1 -- 172.21.15.46:0/622792046 wait complete. 2024-04-26T21:32:49.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.010+0000 7f2277c47640 1 Processor -- start 2024-04-26T21:32:49.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.010+0000 7f2277c47640 1 -- start start 2024-04-26T21:32:49.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.011+0000 7f2277c47640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2270103470 0x7f2270198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:49.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.011+0000 7f2277c47640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2270105da0 0x7f2270199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:49.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.011+0000 7f2277c47640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22701086d0 0x7f22701a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:49.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.011+0000 7f2277c47640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f227010e490 con 0x7f22701086d0 2024-04-26T21:32:49.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.011+0000 7f2277c47640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f227010e310 con 0x7f2270103470 2024-04-26T21:32:49.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.011+0000 7f2277c47640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f227010e610 con 0x7f2270105da0 2024-04-26T21:32:49.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.011+0000 7f22761e4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22701086d0 0x7f22701a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:49.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.011+0000 7f22761e4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22701086d0 0x7f22701a01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49396/0 (socket says 172.21.15.46:49396) 2024-04-26T21:32:49.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.011+0000 7f22761e4640 1 -- 172.21.15.46:0/1940720762 learned_addr learned my addr 172.21.15.46:0/1940720762 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:49.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.011+0000 7f22759e3640 1 --2- 172.21.15.46:0/1940720762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2270103470 0x7f2270198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:49.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.011+0000 7f22751e2640 1 --2- 172.21.15.46:0/1940720762 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2270105da0 0x7f2270199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:49.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.011+0000 7f22761e4640 1 -- 172.21.15.46:0/1940720762 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2270105da0 msgr2=0x7f2270199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:49.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.011+0000 7f22761e4640 1 --2- 172.21.15.46:0/1940720762 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2270105da0 0x7f2270199180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:49.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.011+0000 7f22761e4640 1 -- 172.21.15.46:0/1940720762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2270103470 msgr2=0x7f2270198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:49.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.011+0000 7f22761e4640 1 --2- 172.21.15.46:0/1940720762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2270103470 0x7f2270198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:49.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.011+0000 7f22761e4640 1 -- 172.21.15.46:0/1940720762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22701a0850 con 0x7f22701086d0 2024-04-26T21:32:49.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.011+0000 7f22759e3640 1 --2- 172.21.15.46:0/1940720762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2270103470 0x7f2270198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:32:49.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.011+0000 7f22761e4640 1 --2- 172.21.15.46:0/1940720762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22701086d0 0x7f22701a01b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f226800efd0 tx=0x7f226800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:49.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.011+0000 7f2266ffd640 1 -- 172.21.15.46:0/1940720762 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2268003b70 con 0x7f22701086d0 2024-04-26T21:32:49.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.011+0000 7f2277c47640 1 -- 172.21.15.46:0/1940720762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22701a0b40 con 0x7f22701086d0 2024-04-26T21:32:49.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.011+0000 7f2266ffd640 1 -- 172.21.15.46:0/1940720762 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2268003d10 con 0x7f22701086d0 2024-04-26T21:32:49.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.011+0000 7f2266ffd640 1 -- 172.21.15.46:0/1940720762 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22680108a0 con 0x7f22701086d0 2024-04-26T21:32:49.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.011+0000 7f2277c47640 1 -- 172.21.15.46:0/1940720762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22701a1090 con 0x7f22701086d0 2024-04-26T21:32:49.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.013+0000 7f2277c47640 1 -- 172.21.15.46:0/1940720762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22700fe910 con 0x7f22701086d0 2024-04-26T21:32:49.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.014+0000 7f2266ffd640 1 -- 172.21.15.46:0/1940720762 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2268004020 con 0x7f22701086d0 2024-04-26T21:32:49.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.014+0000 7f2266ffd640 1 --2- 172.21.15.46:0/1940720762 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f225c0a3bb0 0x7f225c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:49.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.014+0000 7f2266ffd640 1 -- 172.21.15.46:0/1940720762 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f22680d07f0 con 0x7f22701086d0 2024-04-26T21:32:49.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.014+0000 7f22759e3640 1 --2- 172.21.15.46:0/1940720762 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f225c0a3bb0 0x7f225c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:49.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.015+0000 7f22759e3640 1 --2- 172.21.15.46:0/1940720762 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f225c0a3bb0 0x7f225c0a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f226c0058c0 tx=0x7f226c00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:49.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.019+0000 7f2266ffd640 1 -- 172.21.15.46:0/1940720762 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f226809d320 con 0x7f22701086d0 2024-04-26T21:32:49.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:48 smithi050 ceph-mon[32753]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T21:32:49.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:48 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/2300160823' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T21:32:49.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:48 smithi050 ceph-mon[32753]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T21:32:49.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:48 smithi050 ceph-mon[32753]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T21:32:49.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:48 smithi050 ceph-mon[32753]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-26T21:32:49.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:48 smithi050 ceph-mon[32753]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-26T21:32:49.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:48 smithi050 ceph-mon[32753]: pgmap v294: 97 pgs: 19 creating+peering, 70 active+clean, 8 unknown; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-04-26T21:32:49.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:48 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/2417904786' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T21:32:49.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.246+0000 7f2277c47640 1 -- 172.21.15.46:0/1940720762 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f227002dca0 con 0x7f22701086d0 2024-04-26T21:32:49.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:48 smithi142 ceph-mon[32944]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T21:32:49.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:48 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/2300160823' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T21:32:49.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:48 smithi142 ceph-mon[32944]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T21:32:49.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:48 smithi142 ceph-mon[32944]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T21:32:49.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:48 smithi142 ceph-mon[32944]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-26T21:32:49.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:48 smithi142 ceph-mon[32944]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-26T21:32:49.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:48 smithi142 ceph-mon[32944]: pgmap v294: 97 pgs: 19 creating+peering, 70 active+clean, 8 unknown; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-04-26T21:32:49.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:48 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/2417904786' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T21:32:49.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:48 smithi046 ceph-mon[29201]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T21:32:49.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:48 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2300160823' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T21:32:49.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:48 smithi046 ceph-mon[29201]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T21:32:49.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:48 smithi046 ceph-mon[29201]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T21:32:49.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:48 smithi046 ceph-mon[29201]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-26T21:32:49.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:48 smithi046 ceph-mon[29201]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-26T21:32:49.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:48 smithi046 ceph-mon[29201]: pgmap v294: 97 pgs: 19 creating+peering, 70 active+clean, 8 unknown; 577 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-04-26T21:32:49.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:48 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2417904786' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T21:32:49.618 INFO:journalctl@ceph.mds.b.smithi050.stdout:Apr 26 21:32:49 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-b[63049]: 2024-04-26T21:32:49.265+0000 7fe1f111d700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T21:32:49.618 INFO:journalctl@ceph.mds.b.smithi050.stdout:Apr 26 21:32:49 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-b[63049]: 2024-04-26T21:32:49.265+0000 7fe1f111d700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-26T21:32:49.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.885+0000 7f2266ffd640 1 -- 172.21.15.46:0/1940720762 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v21) ==== 115+0+0 (secure 0 0 0) 0x7f226809cc20 con 0x7f22701086d0 2024-04-26T21:32:49.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.888+0000 7f2277c47640 1 -- 172.21.15.46:0/1940720762 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f225c0a3bb0 msgr2=0x7f225c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:49.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.888+0000 7f2277c47640 1 --2- 172.21.15.46:0/1940720762 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f225c0a3bb0 0x7f225c0a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f226c0058c0 tx=0x7f226c00a2d0 comp rx=0 tx=0).stop 2024-04-26T21:32:49.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.888+0000 7f2277c47640 1 -- 172.21.15.46:0/1940720762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22701086d0 msgr2=0x7f22701a01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:49.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.888+0000 7f2277c47640 1 --2- 172.21.15.46:0/1940720762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22701086d0 0x7f22701a01b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f226800efd0 tx=0x7f226800c6a0 comp rx=0 tx=0).stop 2024-04-26T21:32:49.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.888+0000 7f2277c47640 1 -- 172.21.15.46:0/1940720762 shutdown_connections 2024-04-26T21:32:49.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.888+0000 7f2277c47640 1 --2- 172.21.15.46:0/1940720762 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f225c0a3bb0 0x7f225c0a6070 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:49.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.888+0000 7f2277c47640 1 --2- 172.21.15.46:0/1940720762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22701086d0 0x7f22701a01b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:49.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.888+0000 7f2277c47640 1 --2- 172.21.15.46:0/1940720762 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2270105da0 0x7f2270199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:49.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.888+0000 7f2277c47640 1 --2- 172.21.15.46:0/1940720762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2270103470 0x7f2270198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:49.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.888+0000 7f2277c47640 1 -- 172.21.15.46:0/1940720762 >> 172.21.15.46:0/1940720762 conn(0x7f22700fce20 msgr2=0x7f2270103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:49.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.888+0000 7f2277c47640 1 -- 172.21.15.46:0/1940720762 shutdown_connections 2024-04-26T21:32:49.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:49.888+0000 7f2277c47640 1 -- 172.21.15.46:0/1940720762 wait complete. 2024-04-26T21:32:49.902 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-04-26T21:32:49.903 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-26T21:32:50.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.014+0000 7fdc0c3f7640 1 -- 172.21.15.46:0/3761061969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc04105d40 msgr2=0x7fdc04108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:50.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.014+0000 7fdc0c3f7640 1 --2- 172.21.15.46:0/3761061969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc04105d40 0x7fdc04108130 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8009900 tx=0x7fdbf802a410 comp rx=0 tx=0).stop 2024-04-26T21:32:50.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.014+0000 7fdc0c3f7640 1 -- 172.21.15.46:0/3761061969 shutdown_connections 2024-04-26T21:32:50.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.014+0000 7fdc0c3f7640 1 --2- 172.21.15.46:0/3761061969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc04108670 0x7fdc0410aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.014+0000 7fdc0c3f7640 1 --2- 172.21.15.46:0/3761061969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc04105d40 0x7fdc04108130 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.014+0000 7fdc0c3f7640 1 --2- 172.21.15.46:0/3761061969 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdc04103410 0x7fdc04105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.014+0000 7fdc0c3f7640 1 -- 172.21.15.46:0/3761061969 >> 172.21.15.46:0/3761061969 conn(0x7fdc040fce20 msgr2=0x7fdc040ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:50.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.014+0000 7fdc0c3f7640 1 -- 172.21.15.46:0/3761061969 shutdown_connections 2024-04-26T21:32:50.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.014+0000 7fdc0c3f7640 1 -- 172.21.15.46:0/3761061969 wait complete. 2024-04-26T21:32:50.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.015+0000 7fdc0c3f7640 1 Processor -- start 2024-04-26T21:32:50.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.015+0000 7fdc0c3f7640 1 -- start start 2024-04-26T21:32:50.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.015+0000 7fdc0c3f7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdc04103410 0x7fdc04198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:50.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.015+0000 7fdc0c3f7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc04105d40 0x7fdc041991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:50.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.015+0000 7fdc0c3f7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc04108670 0x7fdc041a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:50.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.015+0000 7fdc0c3f7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdc0410e4d0 con 0x7fdc04105d40 2024-04-26T21:32:50.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.015+0000 7fdc0c3f7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdc0410e350 con 0x7fdc04108670 2024-04-26T21:32:50.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.015+0000 7fdc0c3f7640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fdc0410e650 con 0x7fdc04103410 2024-04-26T21:32:50.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.015+0000 7fdc09992640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc04105d40 0x7fdc041991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:50.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.015+0000 7fdc09992640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc04105d40 0x7fdc041991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49424/0 (socket says 172.21.15.46:49424) 2024-04-26T21:32:50.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.015+0000 7fdc09992640 1 -- 172.21.15.46:0/3848389316 learned_addr learned my addr 172.21.15.46:0/3848389316 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:50.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.015+0000 7fdc0a994640 1 --2- 172.21.15.46:0/3848389316 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc04108670 0x7fdc041a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:50.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.015+0000 7fdc0a193640 1 --2- 172.21.15.46:0/3848389316 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdc04103410 0x7fdc04198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:50.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.016+0000 7fdc09992640 1 -- 172.21.15.46:0/3848389316 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdc04103410 msgr2=0x7fdc04198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:50.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.016+0000 7fdc09992640 1 --2- 172.21.15.46:0/3848389316 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdc04103410 0x7fdc04198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.016+0000 7fdc09992640 1 -- 172.21.15.46:0/3848389316 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc04108670 msgr2=0x7fdc041a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:50.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.016+0000 7fdc09992640 1 --2- 172.21.15.46:0/3848389316 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc04108670 0x7fdc041a01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.016+0000 7fdc09992640 1 -- 172.21.15.46:0/3848389316 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc041a0870 con 0x7fdc04105d40 2024-04-26T21:32:50.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.016+0000 7fdc0a994640 1 --2- 172.21.15.46:0/3848389316 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc04108670 0x7fdc041a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:32:50.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.016+0000 7fdc09992640 1 --2- 172.21.15.46:0/3848389316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc04105d40 0x7fdc041991a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8006010 tx=0x7fdbf80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:50.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.016+0000 7fdc037fe640 1 -- 172.21.15.46:0/3848389316 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf803f030 con 0x7fdc04105d40 2024-04-26T21:32:50.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.016+0000 7fdc0c3f7640 1 -- 172.21.15.46:0/3848389316 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc041a0b00 con 0x7fdc04105d40 2024-04-26T21:32:50.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.016+0000 7fdc037fe640 1 -- 172.21.15.46:0/3848389316 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbf80318f0 con 0x7fdc04105d40 2024-04-26T21:32:50.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.016+0000 7fdc037fe640 1 -- 172.21.15.46:0/3848389316 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8035070 con 0x7fdc04105d40 2024-04-26T21:32:50.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.016+0000 7fdc0c3f7640 1 -- 172.21.15.46:0/3848389316 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc041a1040 con 0x7fdc04105d40 2024-04-26T21:32:50.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.017+0000 7fdc0c3f7640 1 -- 172.21.15.46:0/3848389316 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc040fe910 con 0x7fdc04105d40 2024-04-26T21:32:50.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.018+0000 7fdc037fe640 1 -- 172.21.15.46:0/3848389316 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fdbf8042030 con 0x7fdc04105d40 2024-04-26T21:32:50.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.019+0000 7fdc037fe640 1 --2- 172.21.15.46:0/3848389316 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fdbec0a3bb0 0x7fdbec0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:50.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.019+0000 7fdc0a193640 1 --2- 172.21.15.46:0/3848389316 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fdbec0a3bb0 0x7fdbec0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:50.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.019+0000 7fdc037fe640 1 -- 172.21.15.46:0/3848389316 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fdbf80f0c90 con 0x7fdc04105d40 2024-04-26T21:32:50.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.019+0000 7fdc0a193640 1 --2- 172.21.15.46:0/3848389316 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fdbec0a3bb0 0x7fdbec0a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc005fd0 tx=0x7fdbfc005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:50.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.022+0000 7fdc037fe640 1 -- 172.21.15.46:0/3848389316 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fdbf80bd7c0 con 0x7fdc04105d40 2024-04-26T21:32:50.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.236+0000 7fdc0c3f7640 1 -- 172.21.15.46:0/3848389316 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fdc04062d70 con 0x7fdbec0a3bb0 2024-04-26T21:32:50.242 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:49 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2417904786' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T21:32:50.242 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:49 smithi046 ceph-mon[29201]: mds.? [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] up:active 2024-04-26T21:32:50.242 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:49 smithi046 ceph-mon[29201]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T21:32:50.242 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:49 smithi046 ceph-mon[29201]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T21:32:50.242 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:49 smithi046 ceph-mon[29201]: Cluster is now healthy 2024-04-26T21:32:50.242 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:49 smithi046 ceph-mon[29201]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-26T21:32:50.242 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:49 smithi046 ceph-mon[29201]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-26T21:32:50.242 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:49 smithi046 ceph-mon[29201]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-26T21:32:50.242 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:49 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1940720762' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T21:32:50.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.262+0000 7fdc037fe640 1 -- 172.21.15.46:0/3848389316 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fdc04062d70 con 0x7fdbec0a3bb0 2024-04-26T21:32:50.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.264+0000 7fdc0c3f7640 1 -- 172.21.15.46:0/3848389316 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fdbec0a3bb0 msgr2=0x7fdbec0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:50.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.264+0000 7fdc0c3f7640 1 --2- 172.21.15.46:0/3848389316 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fdbec0a3bb0 0x7fdbec0a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc005fd0 tx=0x7fdbfc005850 comp rx=0 tx=0).stop 2024-04-26T21:32:50.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.264+0000 7fdc0c3f7640 1 -- 172.21.15.46:0/3848389316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc04105d40 msgr2=0x7fdc041991a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:50.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.264+0000 7fdc0c3f7640 1 --2- 172.21.15.46:0/3848389316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc04105d40 0x7fdc041991a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8006010 tx=0x7fdbf80042f0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.264+0000 7fdc0c3f7640 1 -- 172.21.15.46:0/3848389316 shutdown_connections 2024-04-26T21:32:50.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.265+0000 7fdc0c3f7640 1 --2- 172.21.15.46:0/3848389316 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fdbec0a3bb0 0x7fdbec0a6070 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.265+0000 7fdc0c3f7640 1 --2- 172.21.15.46:0/3848389316 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc04108670 0x7fdc041a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.265+0000 7fdc0c3f7640 1 --2- 172.21.15.46:0/3848389316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc04105d40 0x7fdc041991a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.265+0000 7fdc0c3f7640 1 --2- 172.21.15.46:0/3848389316 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdc04103410 0x7fdc04198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.265+0000 7fdc0c3f7640 1 -- 172.21.15.46:0/3848389316 >> 172.21.15.46:0/3848389316 conn(0x7fdc040fce20 msgr2=0x7fdc04105290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:50.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.265+0000 7fdc0c3f7640 1 -- 172.21.15.46:0/3848389316 shutdown_connections 2024-04-26T21:32:50.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.265+0000 7fdc0c3f7640 1 -- 172.21.15.46:0/3848389316 wait complete. 2024-04-26T21:32:50.279 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-26T21:32:50.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:49 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/2417904786' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T21:32:50.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:49 smithi142 ceph-mon[32944]: mds.? [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] up:active 2024-04-26T21:32:50.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:49 smithi142 ceph-mon[32944]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T21:32:50.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:49 smithi142 ceph-mon[32944]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T21:32:50.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:49 smithi142 ceph-mon[32944]: Cluster is now healthy 2024-04-26T21:32:50.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:49 smithi142 ceph-mon[32944]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-26T21:32:50.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:49 smithi142 ceph-mon[32944]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-26T21:32:50.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:49 smithi142 ceph-mon[32944]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-26T21:32:50.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:49 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/1940720762' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T21:32:50.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:49 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/2417904786' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T21:32:50.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:49 smithi050 ceph-mon[32753]: mds.? [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] up:active 2024-04-26T21:32:50.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:49 smithi050 ceph-mon[32753]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T21:32:50.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:49 smithi050 ceph-mon[32753]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T21:32:50.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:49 smithi050 ceph-mon[32753]: Cluster is now healthy 2024-04-26T21:32:50.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:49 smithi050 ceph-mon[32753]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-26T21:32:50.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:49 smithi050 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-04-26T21:32:50.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:49 smithi050 ceph-mon[32753]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-26T21:32:50.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:49 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/1940720762' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T21:32:50.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.386+0000 7f7f1b452640 1 -- 172.21.15.46:0/2003241756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f14106760 msgr2=0x7f7f1410cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:50.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.386+0000 7f7f1b452640 1 --2- 172.21.15.46:0/2003241756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f14106760 0x7f7f1410cff0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c009ca0 tx=0x7f7f0c02a8c0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.386+0000 7f7f1b452640 1 -- 172.21.15.46:0/2003241756 shutdown_connections 2024-04-26T21:32:50.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.386+0000 7f7f1b452640 1 --2- 172.21.15.46:0/2003241756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f14106760 0x7f7f1410cff0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.386+0000 7f7f1b452640 1 --2- 172.21.15.46:0/2003241756 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f14105da0 0x7f7f14106220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.386+0000 7f7f1b452640 1 --2- 172.21.15.46:0/2003241756 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f14104bb0 0x7f7f14104fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.386+0000 7f7f1b452640 1 -- 172.21.15.46:0/2003241756 >> 172.21.15.46:0/2003241756 conn(0x7f7f14100120 msgr2=0x7f7f14102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:50.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.386+0000 7f7f1b452640 1 -- 172.21.15.46:0/2003241756 shutdown_connections 2024-04-26T21:32:50.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.386+0000 7f7f1b452640 1 -- 172.21.15.46:0/2003241756 wait complete. 2024-04-26T21:32:50.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.386+0000 7f7f1b452640 1 Processor -- start 2024-04-26T21:32:50.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.387+0000 7f7f1b452640 1 -- start start 2024-04-26T21:32:50.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.387+0000 7f7f1b452640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f14104bb0 0x7f7f1419ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:50.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.387+0000 7f7f1b452640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f14105da0 0x7f7f1419b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:50.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.387+0000 7f7f1b452640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f14106760 0x7f7f141a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:50.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.387+0000 7f7f1b452640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7f141106e0 con 0x7f7f14104bb0 2024-04-26T21:32:50.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.387+0000 7f7f1b452640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7f14110560 con 0x7f7f14106760 2024-04-26T21:32:50.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.387+0000 7f7f1b452640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7f14110860 con 0x7f7f14105da0 2024-04-26T21:32:50.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.387+0000 7f7f191ee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f14104bb0 0x7f7f1419ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:50.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.387+0000 7f7f191ee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f14104bb0 0x7f7f1419ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49462/0 (socket says 172.21.15.46:49462) 2024-04-26T21:32:50.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.387+0000 7f7f191ee640 1 -- 172.21.15.46:0/4076889541 learned_addr learned my addr 172.21.15.46:0/4076889541 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:50.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.387+0000 7f7f199ef640 1 --2- 172.21.15.46:0/4076889541 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f14106760 0x7f7f141a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:50.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.387+0000 7f7f189ed640 1 --2- 172.21.15.46:0/4076889541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f14105da0 0x7f7f1419b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:50.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.387+0000 7f7f191ee640 1 -- 172.21.15.46:0/4076889541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f14105da0 msgr2=0x7f7f1419b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:50.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.387+0000 7f7f191ee640 1 --2- 172.21.15.46:0/4076889541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f14105da0 0x7f7f1419b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.387+0000 7f7f191ee640 1 -- 172.21.15.46:0/4076889541 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f14106760 msgr2=0x7f7f141a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:50.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.387+0000 7f7f191ee640 1 --2- 172.21.15.46:0/4076889541 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f14106760 0x7f7f141a2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.387+0000 7f7f191ee640 1 -- 172.21.15.46:0/4076889541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f141a29b0 con 0x7f7f14104bb0 2024-04-26T21:32:50.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.387+0000 7f7f191ee640 1 --2- 172.21.15.46:0/4076889541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f14104bb0 0x7f7f1419ada0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7f1000c970 tx=0x7f7f1000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:50.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.387+0000 7f7f0a7fc640 1 -- 172.21.15.46:0/4076889541 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f10004220 con 0x7f7f14104bb0 2024-04-26T21:32:50.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.387+0000 7f7f1b452640 1 -- 172.21.15.46:0/4076889541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f141a2ca0 con 0x7f7f14104bb0 2024-04-26T21:32:50.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.387+0000 7f7f0a7fc640 1 -- 172.21.15.46:0/4076889541 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f100043c0 con 0x7f7f14104bb0 2024-04-26T21:32:50.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.387+0000 7f7f0a7fc640 1 -- 172.21.15.46:0/4076889541 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f10004a00 con 0x7f7f14104bb0 2024-04-26T21:32:50.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.387+0000 7f7f1b452640 1 -- 172.21.15.46:0/4076889541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f141a31a0 con 0x7f7f14104bb0 2024-04-26T21:32:50.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.388+0000 7f7f1b452640 1 -- 172.21.15.46:0/4076889541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f1410bfa0 con 0x7f7f14104bb0 2024-04-26T21:32:50.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.390+0000 7f7f0a7fc640 1 -- 172.21.15.46:0/4076889541 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7f100079b0 con 0x7f7f14104bb0 2024-04-26T21:32:50.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.390+0000 7f7f0a7fc640 1 --2- 172.21.15.46:0/4076889541 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7f000a3bb0 0x7f7f000a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:50.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.390+0000 7f7f189ed640 1 --2- 172.21.15.46:0/4076889541 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7f000a3bb0 0x7f7f000a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:50.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.390+0000 7f7f0a7fc640 1 -- 172.21.15.46:0/4076889541 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f7f100d00a0 con 0x7f7f14104bb0 2024-04-26T21:32:50.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.391+0000 7f7f189ed640 1 --2- 172.21.15.46:0/4076889541 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7f000a3bb0 0x7f7f000a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009b60 tx=0x7f7f04005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:50.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.395+0000 7f7f0a7fc640 1 -- 172.21.15.46:0/4076889541 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7f10014030 con 0x7f7f14104bb0 2024-04-26T21:32:50.553 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:32:50 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:32:50.241+0000 7f944a7a8700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T21:32:50.553 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:32:50 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:32:50.241+0000 7f944a7a8700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T21:32:50.553 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:32:50 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:32:50.241+0000 7f944a7a8700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T21:32:50.554 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:32:50 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:32:50.241+0000 7f944a7a8700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T21:32:50.554 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:32:50 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:32:50.241+0000 7f944a7a8700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T21:32:50.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.610+0000 7f7f1b452640 1 -- 172.21.15.46:0/4076889541 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f7f14062d70 con 0x7f7f000a3bb0 2024-04-26T21:32:50.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.633+0000 7f7f0a7fc640 1 -- 172.21.15.46:0/4076889541 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7f14062d70 con 0x7f7f000a3bb0 2024-04-26T21:32:50.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.636+0000 7f7f1b452640 1 -- 172.21.15.46:0/4076889541 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7f000a3bb0 msgr2=0x7f7f000a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:50.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.636+0000 7f7f1b452640 1 --2- 172.21.15.46:0/4076889541 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7f000a3bb0 0x7f7f000a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009b60 tx=0x7f7f04005f90 comp rx=0 tx=0).stop 2024-04-26T21:32:50.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.636+0000 7f7f1b452640 1 -- 172.21.15.46:0/4076889541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f14104bb0 msgr2=0x7f7f1419ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:50.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.636+0000 7f7f1b452640 1 --2- 172.21.15.46:0/4076889541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f14104bb0 0x7f7f1419ada0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7f1000c970 tx=0x7f7f1000ce30 comp rx=0 tx=0).stop 2024-04-26T21:32:50.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.636+0000 7f7f1b452640 1 -- 172.21.15.46:0/4076889541 shutdown_connections 2024-04-26T21:32:50.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.636+0000 7f7f1b452640 1 --2- 172.21.15.46:0/4076889541 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7f000a3bb0 0x7f7f000a6070 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.636+0000 7f7f1b452640 1 --2- 172.21.15.46:0/4076889541 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f14106760 0x7f7f141a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.636+0000 7f7f1b452640 1 --2- 172.21.15.46:0/4076889541 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7f14105da0 0x7f7f1419b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.636+0000 7f7f1b452640 1 --2- 172.21.15.46:0/4076889541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f14104bb0 0x7f7f1419ada0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.636+0000 7f7f1b452640 1 -- 172.21.15.46:0/4076889541 >> 172.21.15.46:0/4076889541 conn(0x7f7f14100120 msgr2=0x7f7f14109170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:50.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.636+0000 7f7f1b452640 1 -- 172.21.15.46:0/4076889541 shutdown_connections 2024-04-26T21:32:50.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.636+0000 7f7f1b452640 1 -- 172.21.15.46:0/4076889541 wait complete. 2024-04-26T21:32:50.647 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-26T21:32:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.753+0000 7f84a2f89640 1 -- 172.21.15.46:0/3586591581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f849c103440 msgr2=0x7f849c109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.753+0000 7f84a2f89640 1 --2- 172.21.15.46:0/3586591581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f849c103440 0x7f849c109cd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8494009ca0 tx=0x7f849402a8c0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.754+0000 7f84a2f89640 1 -- 172.21.15.46:0/3586591581 shutdown_connections 2024-04-26T21:32:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.754+0000 7f84a2f89640 1 --2- 172.21.15.46:0/3586591581 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f849c103440 0x7f849c109cd0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.754+0000 7f84a2f89640 1 --2- 172.21.15.46:0/3586591581 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f849c102a80 0x7f849c102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.754+0000 7f84a2f89640 1 --2- 172.21.15.46:0/3586591581 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f849c101890 0x7f849c101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.754+0000 7f84a2f89640 1 -- 172.21.15.46:0/3586591581 >> 172.21.15.46:0/3586591581 conn(0x7f849c0fce20 msgr2=0x7f849c0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.754+0000 7f84a2f89640 1 -- 172.21.15.46:0/3586591581 shutdown_connections 2024-04-26T21:32:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.754+0000 7f84a2f89640 1 -- 172.21.15.46:0/3586591581 wait complete. 2024-04-26T21:32:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.754+0000 7f84a2f89640 1 Processor -- start 2024-04-26T21:32:50.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.754+0000 7f84a2f89640 1 -- start start 2024-04-26T21:32:50.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.754+0000 7f84a2f89640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f849c101890 0x7f849c198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:50.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.754+0000 7f84a2f89640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f849c102a80 0x7f849c1990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:50.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.754+0000 7f84a2f89640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f849c103440 0x7f849c1a0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:50.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.754+0000 7f84a2f89640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f849c10e4a0 con 0x7f849c103440 2024-04-26T21:32:50.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.754+0000 7f84a2f89640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f849c10e320 con 0x7f849c101890 2024-04-26T21:32:50.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.754+0000 7f84a2f89640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f849c10e620 con 0x7f849c102a80 2024-04-26T21:32:50.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.754+0000 7f84a1526640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f849c103440 0x7f849c1a0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:50.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.754+0000 7f84a0d25640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f849c101890 0x7f849c198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:50.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.754+0000 7f84a1526640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f849c103440 0x7f849c1a0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49468/0 (socket says 172.21.15.46:49468) 2024-04-26T21:32:50.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.754+0000 7f84a1526640 1 -- 172.21.15.46:0/3981995946 learned_addr learned my addr 172.21.15.46:0/3981995946 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:50.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.755+0000 7f849bfff640 1 --2- 172.21.15.46:0/3981995946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f849c102a80 0x7f849c1990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:50.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.755+0000 7f84a0d25640 1 -- 172.21.15.46:0/3981995946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f849c102a80 msgr2=0x7f849c1990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:50.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.755+0000 7f84a0d25640 1 --2- 172.21.15.46:0/3981995946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f849c102a80 0x7f849c1990d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.755+0000 7f84a0d25640 1 -- 172.21.15.46:0/3981995946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f849c103440 msgr2=0x7f849c1a0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:50.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.755+0000 7f84a0d25640 1 --2- 172.21.15.46:0/3981995946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f849c103440 0x7f849c1a0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.755+0000 7f84a0d25640 1 -- 172.21.15.46:0/3981995946 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f849c1a07b0 con 0x7f849c101890 2024-04-26T21:32:50.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.755+0000 7f84a1526640 1 --2- 172.21.15.46:0/3981995946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f849c103440 0x7f849c1a0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:32:50.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.755+0000 7f84a0d25640 1 --2- 172.21.15.46:0/3981995946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f849c101890 0x7f849c198b90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f849000e9c0 tx=0x7f849000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:50.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.755+0000 7f8499ffb640 1 -- 172.21.15.46:0/3981995946 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f849000cd80 con 0x7f849c101890 2024-04-26T21:32:50.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.755+0000 7f84a2f89640 1 -- 172.21.15.46:0/3981995946 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f849c1a0aa0 con 0x7f849c101890 2024-04-26T21:32:50.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.755+0000 7f8499ffb640 1 -- 172.21.15.46:0/3981995946 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84900108e0 con 0x7f849c101890 2024-04-26T21:32:50.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.755+0000 7f8499ffb640 1 -- 172.21.15.46:0/3981995946 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84900187f0 con 0x7f849c101890 2024-04-26T21:32:50.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.755+0000 7f84a2f89640 1 -- 172.21.15.46:0/3981995946 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f849c1a1040 con 0x7f849c101890 2024-04-26T21:32:50.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.756+0000 7f84a2f89640 1 -- 172.21.15.46:0/3981995946 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f849c108ca0 con 0x7f849c101890 2024-04-26T21:32:50.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.757+0000 7f8499ffb640 1 -- 172.21.15.46:0/3981995946 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f849001f020 con 0x7f849c101890 2024-04-26T21:32:50.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.758+0000 7f8499ffb640 1 --2- 172.21.15.46:0/3981995946 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f848c0a3bb0 0x7f848c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:50.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.758+0000 7f849bfff640 1 --2- 172.21.15.46:0/3981995946 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f848c0a3bb0 0x7f848c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:50.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.758+0000 7f8499ffb640 1 -- 172.21.15.46:0/3981995946 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f84900d0390 con 0x7f849c101890 2024-04-26T21:32:50.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.758+0000 7f849bfff640 1 --2- 172.21.15.46:0/3981995946 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f848c0a3bb0 0x7f848c0a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f849c19a0a0 tx=0x7f848800b590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:50.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.763+0000 7f8499ffb640 1 -- 172.21.15.46:0/3981995946 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f849009ce20 con 0x7f849c101890 2024-04-26T21:32:50.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.987+0000 7f84a2f89640 1 -- 172.21.15.46:0/3981995946 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f849c062e00 con 0x7f848c0a3bb0 2024-04-26T21:32:50.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.992+0000 7f8499ffb640 1 -- 172.21.15.46:0/3981995946 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f849c062e00 con 0x7f848c0a3bb0 2024-04-26T21:32:50.992 INFO:teuthology.orchestra.run.smithi046.stdout:/volumes/qa/sv_0/73f13d76-e87f-4f90-920c-dbea5d2994b7 2024-04-26T21:32:50.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.994+0000 7f84a2f89640 1 -- 172.21.15.46:0/3981995946 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f848c0a3bb0 msgr2=0x7f848c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:50.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.994+0000 7f84a2f89640 1 --2- 172.21.15.46:0/3981995946 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f848c0a3bb0 0x7f848c0a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f849c19a0a0 tx=0x7f848800b590 comp rx=0 tx=0).stop 2024-04-26T21:32:50.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.994+0000 7f84a2f89640 1 -- 172.21.15.46:0/3981995946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f849c101890 msgr2=0x7f849c198b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:50.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.994+0000 7f84a2f89640 1 --2- 172.21.15.46:0/3981995946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f849c101890 0x7f849c198b90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f849000e9c0 tx=0x7f849000ee80 comp rx=0 tx=0).stop 2024-04-26T21:32:50.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.994+0000 7f84a2f89640 1 -- 172.21.15.46:0/3981995946 shutdown_connections 2024-04-26T21:32:50.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.994+0000 7f84a2f89640 1 --2- 172.21.15.46:0/3981995946 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f848c0a3bb0 0x7f848c0a6070 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.994+0000 7f84a2f89640 1 --2- 172.21.15.46:0/3981995946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f849c103440 0x7f849c1a0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.994+0000 7f84a2f89640 1 --2- 172.21.15.46:0/3981995946 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f849c102a80 0x7f849c1990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.994+0000 7f84a2f89640 1 --2- 172.21.15.46:0/3981995946 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f849c101890 0x7f849c198b90 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:50.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.994+0000 7f84a2f89640 1 -- 172.21.15.46:0/3981995946 >> 172.21.15.46:0/3981995946 conn(0x7f849c0fce20 msgr2=0x7f849c105e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:50.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.994+0000 7f84a2f89640 1 -- 172.21.15.46:0/3981995946 shutdown_connections 2024-04-26T21:32:50.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:50.994+0000 7f84a2f89640 1 -- 172.21.15.46:0/3981995946 wait complete. 2024-04-26T21:32:51.004 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:50 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1940720762' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T21:32:51.004 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:50 smithi046 ceph-mon[29201]: mds.? [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] up:active 2024-04-26T21:32:51.004 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:50 smithi046 ceph-mon[29201]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-26T21:32:51.004 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:50 smithi046 ceph-mon[29201]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:32:51.005 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:50 smithi046 ceph-mon[29201]: pgmap v295: 97 pgs: 97 active+clean; 581 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s wr, 13 op/s 2024-04-26T21:32:51.005 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:50 smithi046 ceph-mon[29201]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:32:51.005 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:50 smithi046 ceph-mon[29201]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:32:51.006 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-26T21:32:51.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.113+0000 7f55dfc83640 1 -- 172.21.15.46:0/4286664868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55d8105d40 msgr2=0x7f55d8108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:51.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.113+0000 7f55dfc83640 1 --2- 172.21.15.46:0/4286664868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55d8105d40 0x7f55d8108130 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f55c8009900 tx=0x7f55c802a410 comp rx=0 tx=0).stop 2024-04-26T21:32:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.113+0000 7f55dfc83640 1 -- 172.21.15.46:0/4286664868 shutdown_connections 2024-04-26T21:32:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.113+0000 7f55dfc83640 1 --2- 172.21.15.46:0/4286664868 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55d8108670 0x7f55d810aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.113+0000 7f55dfc83640 1 --2- 172.21.15.46:0/4286664868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55d8105d40 0x7f55d8108130 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.113+0000 7f55dfc83640 1 --2- 172.21.15.46:0/4286664868 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55d8103410 0x7f55d8105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.113+0000 7f55dfc83640 1 -- 172.21.15.46:0/4286664868 >> 172.21.15.46:0/4286664868 conn(0x7f55d80fce20 msgr2=0x7f55d80ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.113+0000 7f55dfc83640 1 -- 172.21.15.46:0/4286664868 shutdown_connections 2024-04-26T21:32:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.113+0000 7f55dfc83640 1 -- 172.21.15.46:0/4286664868 wait complete. 2024-04-26T21:32:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.114+0000 7f55dfc83640 1 Processor -- start 2024-04-26T21:32:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.114+0000 7f55dfc83640 1 -- start start 2024-04-26T21:32:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.114+0000 7f55dfc83640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55d8103410 0x7f55d8102d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.114+0000 7f55dfc83640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55d8105d40 0x7f55d81013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.114+0000 7f55dfc83640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55d8108670 0x7f55d8101900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:51.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.114+0000 7f55dfc83640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f55d810e500 con 0x7f55d8103410 2024-04-26T21:32:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.114+0000 7f55dfc83640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f55d810e380 con 0x7f55d8108670 2024-04-26T21:32:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.114+0000 7f55dfc83640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f55d810e680 con 0x7f55d8105d40 2024-04-26T21:32:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.114+0000 7f55dda1f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55d8103410 0x7f55d8102d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.114+0000 7f55dda1f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55d8103410 0x7f55d8102d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49482/0 (socket says 172.21.15.46:49482) 2024-04-26T21:32:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.114+0000 7f55dda1f640 1 -- 172.21.15.46:0/1369202138 learned_addr learned my addr 172.21.15.46:0/1369202138 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.114+0000 7f55de220640 1 --2- 172.21.15.46:0/1369202138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55d8108670 0x7f55d8101900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.114+0000 7f55dda1f640 1 -- 172.21.15.46:0/1369202138 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55d8105d40 msgr2=0x7f55d81013c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.114+0000 7f55dda1f640 1 --2- 172.21.15.46:0/1369202138 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55d8105d40 0x7f55d81013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.114+0000 7f55dda1f640 1 -- 172.21.15.46:0/1369202138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55d8108670 msgr2=0x7f55d8101900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.114+0000 7f55dda1f640 1 --2- 172.21.15.46:0/1369202138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55d8108670 0x7f55d8101900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.114+0000 7f55dda1f640 1 -- 172.21.15.46:0/1369202138 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55d8102190 con 0x7f55d8103410 2024-04-26T21:32:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.114+0000 7f55de220640 1 --2- 172.21.15.46:0/1369202138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55d8108670 0x7f55d8101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:32:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.114+0000 7f55dda1f640 1 --2- 172.21.15.46:0/1369202138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55d8103410 0x7f55d8102d80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f55d400d950 tx=0x7f55d400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.115+0000 7f55ceffd640 1 -- 172.21.15.46:0/1369202138 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d4017070 con 0x7f55d8103410 2024-04-26T21:32:51.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.115+0000 7f55ceffd640 1 -- 172.21.15.46:0/1369202138 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55d4009c30 con 0x7f55d8103410 2024-04-26T21:32:51.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.115+0000 7f55dfc83640 1 -- 172.21.15.46:0/1369202138 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55d81a2ca0 con 0x7f55d8103410 2024-04-26T21:32:51.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.115+0000 7f55ceffd640 1 -- 172.21.15.46:0/1369202138 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d400f620 con 0x7f55d8103410 2024-04-26T21:32:51.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.115+0000 7f55dfc83640 1 -- 172.21.15.46:0/1369202138 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55d81a31a0 con 0x7f55d8103410 2024-04-26T21:32:51.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.116+0000 7f55dfc83640 1 -- 172.21.15.46:0/1369202138 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55d80fe910 con 0x7f55d8103410 2024-04-26T21:32:51.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.117+0000 7f55ceffd640 1 -- 172.21.15.46:0/1369202138 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f55d4010460 con 0x7f55d8103410 2024-04-26T21:32:51.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.118+0000 7f55ceffd640 1 --2- 172.21.15.46:0/1369202138 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f55c40a3bb0 0x7f55c40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:51.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.118+0000 7f55dd21e640 1 --2- 172.21.15.46:0/1369202138 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f55c40a3bb0 0x7f55c40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:51.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.118+0000 7f55ceffd640 1 -- 172.21.15.46:0/1369202138 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f55d40cffb0 con 0x7f55d8103410 2024-04-26T21:32:51.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.118+0000 7f55dd21e640 1 --2- 172.21.15.46:0/1369202138 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f55c40a3bb0 0x7f55c40a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f55c8009230 tx=0x7f55c8000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:51.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.122+0000 7f55ceffd640 1 -- 172.21.15.46:0/1369202138 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f55d409cae0 con 0x7f55d8103410 2024-04-26T21:32:51.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:50 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/1940720762' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T21:32:51.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:50 smithi142 ceph-mon[32944]: mds.? [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] up:active 2024-04-26T21:32:51.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:50 smithi142 ceph-mon[32944]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-26T21:32:51.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:50 smithi142 ceph-mon[32944]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:32:51.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:50 smithi142 ceph-mon[32944]: pgmap v295: 97 pgs: 97 active+clean; 581 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s wr, 13 op/s 2024-04-26T21:32:51.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:50 smithi142 ceph-mon[32944]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:32:51.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:50 smithi142 ceph-mon[32944]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:32:51.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.337+0000 7f55dfc83640 1 -- 172.21.15.46:0/1369202138 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f55d8062e00 con 0x7f55c40a3bb0 2024-04-26T21:32:51.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.366+0000 7f55ceffd640 1 -- 172.21.15.46:0/1369202138 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f55d8062e00 con 0x7f55c40a3bb0 2024-04-26T21:32:51.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:50 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/1940720762' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T21:32:51.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:50 smithi050 ceph-mon[32753]: mds.? [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] up:active 2024-04-26T21:32:51.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:50 smithi050 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-04-26T21:32:51.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:50 smithi050 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:32:51.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:50 smithi050 ceph-mon[32753]: pgmap v295: 97 pgs: 97 active+clean; 581 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s wr, 13 op/s 2024-04-26T21:32:51.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:50 smithi050 ceph-mon[32753]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:32:51.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:50 smithi050 ceph-mon[32753]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:32:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.368+0000 7f55dfc83640 1 -- 172.21.15.46:0/1369202138 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f55c40a3bb0 msgr2=0x7f55c40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.368+0000 7f55dfc83640 1 --2- 172.21.15.46:0/1369202138 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f55c40a3bb0 0x7f55c40a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f55c8009230 tx=0x7f55c8000bc0 comp rx=0 tx=0).stop 2024-04-26T21:32:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.368+0000 7f55dfc83640 1 -- 172.21.15.46:0/1369202138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55d8103410 msgr2=0x7f55d8102d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.368+0000 7f55dfc83640 1 --2- 172.21.15.46:0/1369202138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55d8103410 0x7f55d8102d80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f55d400d950 tx=0x7f55d400de10 comp rx=0 tx=0).stop 2024-04-26T21:32:51.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.368+0000 7f55dfc83640 1 -- 172.21.15.46:0/1369202138 shutdown_connections 2024-04-26T21:32:51.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.368+0000 7f55dfc83640 1 --2- 172.21.15.46:0/1369202138 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f55c40a3bb0 0x7f55c40a6070 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:51.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.368+0000 7f55dfc83640 1 --2- 172.21.15.46:0/1369202138 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f55d8108670 0x7f55d8101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:51.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.368+0000 7f55dfc83640 1 --2- 172.21.15.46:0/1369202138 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f55d8105d40 0x7f55d81013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:51.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.368+0000 7f55dfc83640 1 --2- 172.21.15.46:0/1369202138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f55d8103410 0x7f55d8102d80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:51.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.368+0000 7f55dfc83640 1 -- 172.21.15.46:0/1369202138 >> 172.21.15.46:0/1369202138 conn(0x7f55d80fce20 msgr2=0x7f55d8103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:51.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.368+0000 7f55dfc83640 1 -- 172.21.15.46:0/1369202138 shutdown_connections 2024-04-26T21:32:51.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.368+0000 7f55dfc83640 1 -- 172.21.15.46:0/1369202138 wait complete. 2024-04-26T21:32:51.380 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-26T21:32:51.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.485+0000 7f5b5ff14640 1 -- 172.21.15.46:0/3388507398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b58102a80 msgr2=0x7f5b58102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:51.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.485+0000 7f5b5ff14640 1 --2- 172.21.15.46:0/3388507398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b58102a80 0x7f5b58102f00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5b48009900 tx=0x7f5b4802a410 comp rx=0 tx=0).stop 2024-04-26T21:32:51.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.485+0000 7f5b5ff14640 1 -- 172.21.15.46:0/3388507398 shutdown_connections 2024-04-26T21:32:51.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.485+0000 7f5b5ff14640 1 --2- 172.21.15.46:0/3388507398 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b58103440 0x7f5b58109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:51.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.486+0000 7f5b5ff14640 1 --2- 172.21.15.46:0/3388507398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b58102a80 0x7f5b58102f00 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:51.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.486+0000 7f5b5ff14640 1 --2- 172.21.15.46:0/3388507398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b58101890 0x7f5b58101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:51.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.486+0000 7f5b5ff14640 1 -- 172.21.15.46:0/3388507398 >> 172.21.15.46:0/3388507398 conn(0x7f5b580fce00 msgr2=0x7f5b580ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:51.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.486+0000 7f5b5ff14640 1 -- 172.21.15.46:0/3388507398 shutdown_connections 2024-04-26T21:32:51.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.486+0000 7f5b5ff14640 1 -- 172.21.15.46:0/3388507398 wait complete. 2024-04-26T21:32:51.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.486+0000 7f5b5ff14640 1 Processor -- start 2024-04-26T21:32:51.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.486+0000 7f5b5ff14640 1 -- start start 2024-04-26T21:32:51.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.486+0000 7f5b5ff14640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b58101890 0x7f5b58197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:51.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.486+0000 7f5b5ff14640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b58102a80 0x7f5b581980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:51.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.486+0000 7f5b5ff14640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b58103440 0x7f5b5819f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:51.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.486+0000 7f5b5ff14640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5b5810d420 con 0x7f5b58103440 2024-04-26T21:32:51.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.486+0000 7f5b5ff14640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5b5810d2a0 con 0x7f5b58101890 2024-04-26T21:32:51.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.486+0000 7f5b5ff14640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5b5810d5a0 con 0x7f5b58102a80 2024-04-26T21:32:51.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.486+0000 7f5b5e4b1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b58103440 0x7f5b5819f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:51.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.486+0000 7f5b5dcb0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b58101890 0x7f5b58197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:51.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.486+0000 7f5b5e4b1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b58103440 0x7f5b5819f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49502/0 (socket says 172.21.15.46:49502) 2024-04-26T21:32:51.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.487+0000 7f5b5e4b1640 1 -- 172.21.15.46:0/4062961445 learned_addr learned my addr 172.21.15.46:0/4062961445 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:51.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.487+0000 7f5b5d4af640 1 --2- 172.21.15.46:0/4062961445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b58102a80 0x7f5b581980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:51.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.487+0000 7f5b5e4b1640 1 -- 172.21.15.46:0/4062961445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b58102a80 msgr2=0x7f5b581980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:51.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.487+0000 7f5b5e4b1640 1 --2- 172.21.15.46:0/4062961445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b58102a80 0x7f5b581980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:51.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.487+0000 7f5b5e4b1640 1 -- 172.21.15.46:0/4062961445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b58101890 msgr2=0x7f5b58197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:51.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.487+0000 7f5b5e4b1640 1 --2- 172.21.15.46:0/4062961445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b58101890 0x7f5b58197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:51.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.487+0000 7f5b5e4b1640 1 -- 172.21.15.46:0/4062961445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b5819f790 con 0x7f5b58103440 2024-04-26T21:32:51.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.487+0000 7f5b5e4b1640 1 --2- 172.21.15.46:0/4062961445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b58103440 0x7f5b5819f120 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5b5000b970 tx=0x7f5b5000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:51.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.487+0000 7f5b4effd640 1 -- 172.21.15.46:0/4062961445 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b50019070 con 0x7f5b58103440 2024-04-26T21:32:51.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.487+0000 7f5b4effd640 1 -- 172.21.15.46:0/4062961445 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b50014440 con 0x7f5b58103440 2024-04-26T21:32:51.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.487+0000 7f5b5ff14640 1 -- 172.21.15.46:0/4062961445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b5819fa80 con 0x7f5b58103440 2024-04-26T21:32:51.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.487+0000 7f5b4effd640 1 -- 172.21.15.46:0/4062961445 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b500075d0 con 0x7f5b58103440 2024-04-26T21:32:51.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.487+0000 7f5b5ff14640 1 -- 172.21.15.46:0/4062961445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b5819ffd0 con 0x7f5b58103440 2024-04-26T21:32:51.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.488+0000 7f5b5ff14640 1 -- 172.21.15.46:0/4062961445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b58108c80 con 0x7f5b58103440 2024-04-26T21:32:51.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.489+0000 7f5b4effd640 1 -- 172.21.15.46:0/4062961445 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5b5001c070 con 0x7f5b58103440 2024-04-26T21:32:51.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.490+0000 7f5b4effd640 1 --2- 172.21.15.46:0/4062961445 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5b440a3bb0 0x7f5b440a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:51.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.490+0000 7f5b5dcb0640 1 --2- 172.21.15.46:0/4062961445 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5b440a3bb0 0x7f5b440a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:51.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.490+0000 7f5b4effd640 1 -- 172.21.15.46:0/4062961445 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f5b500cf800 con 0x7f5b58103440 2024-04-26T21:32:51.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.490+0000 7f5b5dcb0640 1 --2- 172.21.15.46:0/4062961445 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5b440a3bb0 0x7f5b440a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5b581028e0 tx=0x7f5b5400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:51.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.494+0000 7f5b4effd640 1 -- 172.21.15.46:0/4062961445 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5b5009bce0 con 0x7f5b58103440 2024-04-26T21:32:51.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.710+0000 7f5b5ff14640 1 -- 172.21.15.46:0/4062961445 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f5b58062d70 con 0x7f5b440a3bb0 2024-04-26T21:32:51.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.717+0000 7f5b4effd640 1 -- 172.21.15.46:0/4062961445 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f5b58062d70 con 0x7f5b440a3bb0 2024-04-26T21:32:51.718 INFO:teuthology.orchestra.run.smithi046.stdout:/volumes/qa/sv_1/1453fde9-3580-415d-8a6d-8b422bc1e71d 2024-04-26T21:32:51.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.719+0000 7f5b5ff14640 1 -- 172.21.15.46:0/4062961445 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5b440a3bb0 msgr2=0x7f5b440a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:51.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.719+0000 7f5b5ff14640 1 --2- 172.21.15.46:0/4062961445 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5b440a3bb0 0x7f5b440a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5b581028e0 tx=0x7f5b5400b390 comp rx=0 tx=0).stop 2024-04-26T21:32:51.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.719+0000 7f5b5ff14640 1 -- 172.21.15.46:0/4062961445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b58103440 msgr2=0x7f5b5819f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:51.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.719+0000 7f5b5ff14640 1 --2- 172.21.15.46:0/4062961445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b58103440 0x7f5b5819f120 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5b5000b970 tx=0x7f5b5000be30 comp rx=0 tx=0).stop 2024-04-26T21:32:51.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.719+0000 7f5b5ff14640 1 -- 172.21.15.46:0/4062961445 shutdown_connections 2024-04-26T21:32:51.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.719+0000 7f5b5ff14640 1 --2- 172.21.15.46:0/4062961445 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5b440a3bb0 0x7f5b440a6070 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:51.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.719+0000 7f5b5ff14640 1 --2- 172.21.15.46:0/4062961445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b58103440 0x7f5b5819f120 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:51.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.719+0000 7f5b5ff14640 1 --2- 172.21.15.46:0/4062961445 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5b58102a80 0x7f5b581980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:51.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.719+0000 7f5b5ff14640 1 --2- 172.21.15.46:0/4062961445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5b58101890 0x7f5b58197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:51.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.719+0000 7f5b5ff14640 1 -- 172.21.15.46:0/4062961445 >> 172.21.15.46:0/4062961445 conn(0x7f5b580fce00 msgr2=0x7f5b58105e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:51.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.719+0000 7f5b5ff14640 1 -- 172.21.15.46:0/4062961445 shutdown_connections 2024-04-26T21:32:51.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.719+0000 7f5b5ff14640 1 -- 172.21.15.46:0/4062961445 wait complete. 2024-04-26T21:32:51.731 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:32:51.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.831+0000 7f6b4dbe7640 1 -- 172.21.15.46:0/3015908934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b481029a0 msgr2=0x7f6b48102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:51.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.831+0000 7f6b4dbe7640 1 --2- 172.21.15.46:0/3015908934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b481029a0 0x7f6b48102dc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6b40009ca0 tx=0x7f6b4002a8c0 comp rx=0 tx=0).stop 2024-04-26T21:32:51.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.831+0000 7f6b4dbe7640 1 -- 172.21.15.46:0/3015908934 shutdown_connections 2024-04-26T21:32:51.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.831+0000 7f6b4dbe7640 1 --2- 172.21.15.46:0/3015908934 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6b48104550 0x7f6b4810ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:51.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.831+0000 7f6b4dbe7640 1 --2- 172.21.15.46:0/3015908934 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b48103b90 0x7f6b48104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:51.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.831+0000 7f6b4dbe7640 1 --2- 172.21.15.46:0/3015908934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b481029a0 0x7f6b48102dc0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:51.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.831+0000 7f6b4dbe7640 1 -- 172.21.15.46:0/3015908934 >> 172.21.15.46:0/3015908934 conn(0x7f6b480fdf30 msgr2=0x7f6b48100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:51.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.831+0000 7f6b4dbe7640 1 -- 172.21.15.46:0/3015908934 shutdown_connections 2024-04-26T21:32:51.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.831+0000 7f6b4dbe7640 1 -- 172.21.15.46:0/3015908934 wait complete. 2024-04-26T21:32:51.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.832+0000 7f6b4dbe7640 1 Processor -- start 2024-04-26T21:32:51.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.832+0000 7f6b4dbe7640 1 -- start start 2024-04-26T21:32:51.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.832+0000 7f6b4dbe7640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6b481029a0 0x7f6b48198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:51.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.832+0000 7f6b4dbe7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b48103b90 0x7f6b48199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:51.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.832+0000 7f6b4dbe7640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b48104550 0x7f6b481a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:51.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.832+0000 7f6b4dbe7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4810e4d0 con 0x7f6b48103b90 2024-04-26T21:32:51.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.832+0000 7f6b4dbe7640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4810e350 con 0x7f6b48104550 2024-04-26T21:32:51.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.832+0000 7f6b4dbe7640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4810e650 con 0x7f6b481029a0 2024-04-26T21:32:51.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.832+0000 7f6b46ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b48103b90 0x7f6b48199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:51.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.832+0000 7f6b46ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b48103b90 0x7f6b48199160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49520/0 (socket says 172.21.15.46:49520) 2024-04-26T21:32:51.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.832+0000 7f6b46ffd640 1 -- 172.21.15.46:0/3166015083 learned_addr learned my addr 172.21.15.46:0/3166015083 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:51.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.832+0000 7f6b47fff640 1 --2- 172.21.15.46:0/3166015083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b48104550 0x7f6b481a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:51.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.833+0000 7f6b477fe640 1 --2- 172.21.15.46:0/3166015083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6b481029a0 0x7f6b48198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:51.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.833+0000 7f6b46ffd640 1 -- 172.21.15.46:0/3166015083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6b481029a0 msgr2=0x7f6b48198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:51.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.833+0000 7f6b46ffd640 1 --2- 172.21.15.46:0/3166015083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6b481029a0 0x7f6b48198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:51.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.833+0000 7f6b46ffd640 1 -- 172.21.15.46:0/3166015083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b48104550 msgr2=0x7f6b481a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:51.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.833+0000 7f6b46ffd640 1 --2- 172.21.15.46:0/3166015083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b48104550 0x7f6b481a0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:51.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.833+0000 7f6b46ffd640 1 -- 172.21.15.46:0/3166015083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b481a0830 con 0x7f6b48103b90 2024-04-26T21:32:51.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.833+0000 7f6b46ffd640 1 --2- 172.21.15.46:0/3166015083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b48103b90 0x7f6b48199160 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6b3000d950 tx=0x7f6b3000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:51.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.833+0000 7f6b44ff9640 1 -- 172.21.15.46:0/3166015083 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b30017030 con 0x7f6b48103b90 2024-04-26T21:32:51.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.833+0000 7f6b44ff9640 1 -- 172.21.15.46:0/3166015083 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b30004020 con 0x7f6b48103b90 2024-04-26T21:32:51.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.833+0000 7f6b4dbe7640 1 -- 172.21.15.46:0/3166015083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b481a0b20 con 0x7f6b48103b90 2024-04-26T21:32:51.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.833+0000 7f6b44ff9640 1 -- 172.21.15.46:0/3166015083 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b300047a0 con 0x7f6b48103b90 2024-04-26T21:32:51.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.833+0000 7f6b4dbe7640 1 -- 172.21.15.46:0/3166015083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b481a1070 con 0x7f6b48103b90 2024-04-26T21:32:51.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.834+0000 7f6b4dbe7640 1 -- 172.21.15.46:0/3166015083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b48109db0 con 0x7f6b48103b90 2024-04-26T21:32:51.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.835+0000 7f6b44ff9640 1 -- 172.21.15.46:0/3166015083 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6b30009a30 con 0x7f6b48103b90 2024-04-26T21:32:51.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.836+0000 7f6b44ff9640 1 --2- 172.21.15.46:0/3166015083 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6b340a3bb0 0x7f6b340a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:51.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.836+0000 7f6b477fe640 1 --2- 172.21.15.46:0/3166015083 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6b340a3bb0 0x7f6b340a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:51.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.836+0000 7f6b44ff9640 1 -- 172.21.15.46:0/3166015083 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f6b300cfe00 con 0x7f6b48103b90 2024-04-26T21:32:51.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.836+0000 7f6b477fe640 1 --2- 172.21.15.46:0/3166015083 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6b340a3bb0 0x7f6b340a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6b40031b50 tx=0x7f6b40009360 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:51.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:51.841+0000 7f6b44ff9640 1 -- 172.21.15.46:0/3166015083 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6b3009c930 con 0x7f6b48103b90 2024-04-26T21:32:52.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:51 smithi046 ceph-mon[29201]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:32:52.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:51 smithi046 ceph-mon[29201]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:32:52.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:51 smithi046 ceph-mon[29201]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:32:52.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.070+0000 7f6b4dbe7640 1 -- 172.21.15.46:0/3166015083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b4802dad0 con 0x7f6b48103b90 2024-04-26T21:32:52.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.071+0000 7f6b44ff9640 1 -- 172.21.15.46:0/3166015083 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11863 (secure 0 0 0) 0x7f6b3009c230 con 0x7f6b48103b90 2024-04-26T21:32:52.072 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:32:52.072 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3674296286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3674296286},{"type":"v1","addr":"172.21.15.46:6835","nonce":3674296286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/391791206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":391791206},{"type":"v1","addr":"172.21.15.46:6837","nonce":391791206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/1222489912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1222489912},{"type":"v1","addr":"172.21.15.46:6839","nonce":1222489912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14658,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/860605548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":860605548},{"type":"v1","addr":"172.21.15.46:6841","nonce":860605548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3621399933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3621399933},{"type":"v1","addr":"172.21.15.142:6833","nonce":3621399933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/345015778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":345015778},{"type":"v1","addr":"172.21.15.50:6833","nonce":345015778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T21:32:44.201541+0000","modified":"2024-04-26T21:32:49.889029+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24526,"mds_2":24509},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/2546509863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":2546509863},{"type":"v1","addr":"172.21.15.50:6835","nonce":2546509863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/3791468066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":3791468066},{"type":"v1","addr":"172.21.15.142:6835","nonce":3791468066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6837/1723132553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1723132553},{"type":"v1","addr":"172.21.15.142:6837","nonce":1723132553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6837/3189620395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3189620395},{"type":"v1","addr":"172.21.15.50:6837","nonce":3189620395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24526":{"gid":24526,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.142:6839/3089509101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3089509101},{"type":"v1","addr":"172.21.15.142:6839","nonce":3089509101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6839/2453253368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2453253368},{"type":"v1","addr":"172.21.15.50:6839","nonce":2453253368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24509,24526,24539]},"id":1}]} 2024-04-26T21:32:52.072 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 22 2024-04-26T21:32:52.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.073+0000 7f6b4dbe7640 1 -- 172.21.15.46:0/3166015083 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6b340a3bb0 msgr2=0x7f6b340a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:52.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.073+0000 7f6b4dbe7640 1 --2- 172.21.15.46:0/3166015083 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6b340a3bb0 0x7f6b340a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6b40031b50 tx=0x7f6b40009360 comp rx=0 tx=0).stop 2024-04-26T21:32:52.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.073+0000 7f6b4dbe7640 1 -- 172.21.15.46:0/3166015083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b48103b90 msgr2=0x7f6b48199160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:52.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.073+0000 7f6b4dbe7640 1 --2- 172.21.15.46:0/3166015083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b48103b90 0x7f6b48199160 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6b3000d950 tx=0x7f6b3000de10 comp rx=0 tx=0).stop 2024-04-26T21:32:52.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.073+0000 7f6b4dbe7640 1 -- 172.21.15.46:0/3166015083 shutdown_connections 2024-04-26T21:32:52.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.073+0000 7f6b4dbe7640 1 --2- 172.21.15.46:0/3166015083 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6b340a3bb0 0x7f6b340a6070 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.073+0000 7f6b4dbe7640 1 --2- 172.21.15.46:0/3166015083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b48104550 0x7f6b481a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.073+0000 7f6b4dbe7640 1 --2- 172.21.15.46:0/3166015083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b48103b90 0x7f6b48199160 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.073+0000 7f6b4dbe7640 1 --2- 172.21.15.46:0/3166015083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6b481029a0 0x7f6b48198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.073+0000 7f6b4dbe7640 1 -- 172.21.15.46:0/3166015083 >> 172.21.15.46:0/3166015083 conn(0x7f6b480fdf30 msgr2=0x7f6b48106f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:52.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.073+0000 7f6b4dbe7640 1 -- 172.21.15.46:0/3166015083 shutdown_connections 2024-04-26T21:32:52.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.073+0000 7f6b4dbe7640 1 -- 172.21.15.46:0/3166015083 wait complete. 2024-04-26T21:32:52.085 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:32:52.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:51 smithi050 ceph-mon[32753]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:32:52.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:51 smithi050 ceph-mon[32753]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:32:52.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:51 smithi050 ceph-mon[32753]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:32:52.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.185+0000 7eff5b319640 1 -- 172.21.15.46:0/2252387199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff54101260 msgr2=0x7eff54101680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:52.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.185+0000 7eff5b319640 1 --2- 172.21.15.46:0/2252387199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff54101260 0x7eff54101680 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7eff4c009900 tx=0x7eff4c02a480 comp rx=0 tx=0).stop 2024-04-26T21:32:52.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.185+0000 7eff5b319640 1 -- 172.21.15.46:0/2252387199 shutdown_connections 2024-04-26T21:32:52.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.185+0000 7eff5b319640 1 --2- 172.21.15.46:0/2252387199 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff54102560 0x7eff541029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.185+0000 7eff5b319640 1 --2- 172.21.15.46:0/2252387199 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7eff54101bc0 0x7eff54102020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.185+0000 7eff5b319640 1 --2- 172.21.15.46:0/2252387199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff54101260 0x7eff54101680 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.185+0000 7eff5b319640 1 -- 172.21.15.46:0/2252387199 >> 172.21.15.46:0/2252387199 conn(0x7eff540fcde0 msgr2=0x7eff540ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:52.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.185+0000 7eff5b319640 1 -- 172.21.15.46:0/2252387199 shutdown_connections 2024-04-26T21:32:52.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.186+0000 7eff5b319640 1 -- 172.21.15.46:0/2252387199 wait complete. 2024-04-26T21:32:52.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.186+0000 7eff5b319640 1 Processor -- start 2024-04-26T21:32:52.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.186+0000 7eff5b319640 1 -- start start 2024-04-26T21:32:52.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.186+0000 7eff5b319640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7eff54101260 0x7eff5419ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:52.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.186+0000 7eff5b319640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff54101bc0 0x7eff5419b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:52.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.186+0000 7eff5b319640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff54102560 0x7eff541a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:52.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.186+0000 7eff5b319640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7eff54110670 con 0x7eff54102560 2024-04-26T21:32:52.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.186+0000 7eff5b319640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7eff541104f0 con 0x7eff54101bc0 2024-04-26T21:32:52.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.186+0000 7eff5b319640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7eff541107f0 con 0x7eff54101260 2024-04-26T21:32:52.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.186+0000 7eff598b6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff54102560 0x7eff541a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:52.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.186+0000 7eff598b6640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff54102560 0x7eff541a2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49536/0 (socket says 172.21.15.46:49536) 2024-04-26T21:32:52.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.186+0000 7eff598b6640 1 -- 172.21.15.46:0/1925659073 learned_addr learned my addr 172.21.15.46:0/1925659073 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:52.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.186+0000 7eff590b5640 1 --2- 172.21.15.46:0/1925659073 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7eff54101260 0x7eff5419ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:52.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.187+0000 7eff588b4640 1 --2- 172.21.15.46:0/1925659073 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff54101bc0 0x7eff5419b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:52.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.187+0000 7eff598b6640 1 -- 172.21.15.46:0/1925659073 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7eff54101260 msgr2=0x7eff5419ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:52.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.187+0000 7eff598b6640 1 --2- 172.21.15.46:0/1925659073 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7eff54101260 0x7eff5419ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.187+0000 7eff598b6640 1 -- 172.21.15.46:0/1925659073 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff54101bc0 msgr2=0x7eff5419b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:52.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.187+0000 7eff598b6640 1 --2- 172.21.15.46:0/1925659073 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff54101bc0 0x7eff5419b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.187+0000 7eff598b6640 1 -- 172.21.15.46:0/1925659073 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff541a2990 con 0x7eff54102560 2024-04-26T21:32:52.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.187+0000 7eff598b6640 1 --2- 172.21.15.46:0/1925659073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff54102560 0x7eff541a2320 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7eff44009fd0 tx=0x7eff4400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:52.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.187+0000 7eff527fc640 1 -- 172.21.15.46:0/1925659073 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff44017070 con 0x7eff54102560 2024-04-26T21:32:52.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.187+0000 7eff527fc640 1 -- 172.21.15.46:0/1925659073 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff44003e20 con 0x7eff54102560 2024-04-26T21:32:52.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.187+0000 7eff5b319640 1 -- 172.21.15.46:0/1925659073 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff541a2c80 con 0x7eff54102560 2024-04-26T21:32:52.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.187+0000 7eff527fc640 1 -- 172.21.15.46:0/1925659073 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff44010460 con 0x7eff54102560 2024-04-26T21:32:52.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.187+0000 7eff5b319640 1 -- 172.21.15.46:0/1925659073 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff541a3180 con 0x7eff54102560 2024-04-26T21:32:52.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.188+0000 7eff5b319640 1 -- 172.21.15.46:0/1925659073 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff5410c1e0 con 0x7eff54102560 2024-04-26T21:32:52.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.189+0000 7eff527fc640 1 -- 172.21.15.46:0/1925659073 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7eff44003970 con 0x7eff54102560 2024-04-26T21:32:52.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.190+0000 7eff527fc640 1 --2- 172.21.15.46:0/1925659073 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7eff3c0a3bb0 0x7eff3c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:52.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.190+0000 7eff590b5640 1 --2- 172.21.15.46:0/1925659073 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7eff3c0a3bb0 0x7eff3c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:52.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.190+0000 7eff527fc640 1 -- 172.21.15.46:0/1925659073 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7eff440d0010 con 0x7eff54102560 2024-04-26T21:32:52.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.190+0000 7eff590b5640 1 --2- 172.21.15.46:0/1925659073 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7eff3c0a3bb0 0x7eff3c0a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7eff4c000c00 tx=0x7eff4c0314a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:52.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.194+0000 7eff527fc640 1 -- 172.21.15.46:0/1925659073 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7eff4409cb40 con 0x7eff54102560 2024-04-26T21:32:52.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:51 smithi142 ceph-mon[32944]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:32:52.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:51 smithi142 ceph-mon[32944]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:32:52.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:51 smithi142 ceph-mon[32944]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:32:52.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.380+0000 7eff5b319640 1 -- 172.21.15.46:0/1925659073 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7eff540635d0 con 0x7eff54102560 2024-04-26T21:32:52.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.382+0000 7eff527fc640 1 -- 172.21.15.46:0/1925659073 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22197 (secure 0 0 0) 0x7eff4409c440 con 0x7eff54102560 2024-04-26T21:32:52.383 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:32:52.383 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":85,"fsid":"36c7013a-0413-11ef-bc93-c7b262605968","created":"2024-04-26T21:24:30.481156+0000","modified":"2024-04-26T21:32:45.847101+0000","last_up_change":"2024-04-26T21:31:50.100491+0000","last_in_change":"2024-04-26T21:31:29.643428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:27:57.935988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:32:42.195211+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:32:43.162114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f53c8702-e4da-435f-a9dc-3911d0d36ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6803","nonce":2762665125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6805","nonce":2762665125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6809","nonce":2762665125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6807","nonce":2762665125}]},"public_addr":"172.21.15.46:6803/2762665125","cluster_addr":"172.21.15.46:6805/2762665125","heartbeat_back_addr":"172.21.15.46:6809/2762665125","heartbeat_front_addr":"172.21.15.46:6807/2762665125","state":["exists","up"]},{"osd":1,"uuid":"fc4d6f52-fc8a-4efd-80c9-c092b57e7f42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6801","nonce":1628008970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6803","nonce":1628008970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6807","nonce":1628008970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6805","nonce":1628008970}]},"public_addr":"172.21.15.50:6801/1628008970","cluster_addr":"172.21.15.50:6803/1628008970","heartbeat_back_addr":"172.21.15.50:6807/1628008970","heartbeat_front_addr":"172.21.15.50:6805/1628008970","state":["exists","up"]},{"osd":2,"uuid":"26d1baa1-4ea1-4b7e-9fcf-4f8bbf7329c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6801","nonce":1513279137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6803","nonce":1513279137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6807","nonce":1513279137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6805","nonce":1513279137}]},"public_addr":"172.21.15.142:6801/1513279137","cluster_addr":"172.21.15.142:6803/1513279137","heartbeat_back_addr":"172.21.15.142:6807/1513279137","heartbeat_front_addr":"172.21.15.142:6805/1513279137","state":["exists","up"]},{"osd":3,"uuid":"34d2401f-5fcc-491c-8fcf-c3c492611f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6811","nonce":2435181760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6813","nonce":2435181760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6817","nonce":2435181760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6815","nonce":2435181760}]},"public_addr":"172.21.15.46:6811/2435181760","cluster_addr":"172.21.15.46:6813/2435181760","heartbeat_back_addr":"172.21.15.46:6817/2435181760","heartbeat_front_addr":"172.21.15.46:6815/2435181760","state":["exists","up"]},{"osd":4,"uuid":"986bb709-e4cf-400d-982e-c3f5ac413e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6809","nonce":3177333250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6811","nonce":3177333250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6815","nonce":3177333250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6813","nonce":3177333250}]},"public_addr":"172.21.15.50:6809/3177333250","cluster_addr":"172.21.15.50:6811/3177333250","heartbeat_back_addr":"172.21.15.50:6815/3177333250","heartbeat_front_addr":"172.21.15.50:6813/3177333250","state":["exists","up"]},{"osd":5,"uuid":"b7d65851-ebe1-45e2-833d-8022446f2698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6809","nonce":4113107737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6811","nonce":4113107737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6815","nonce":4113107737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6813","nonce":4113107737}]},"public_addr":"172.21.15.142:6809/4113107737","cluster_addr":"172.21.15.142:6811/4113107737","heartbeat_back_addr":"172.21.15.142:6815/4113107737","heartbeat_front_addr":"172.21.15.142:6813/4113107737","state":["exists","up"]},{"osd":6,"uuid":"2a3e96d7-d5bf-44ff-9134-29f1420c0e4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6819","nonce":2325162882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6821","nonce":2325162882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6825","nonce":2325162882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6823","nonce":2325162882}]},"public_addr":"172.21.15.46:6819/2325162882","cluster_addr":"172.21.15.46:6821/2325162882","heartbeat_back_addr":"172.21.15.46:6825/2325162882","heartbeat_front_addr":"172.21.15.46:6823/2325162882","state":["exists","up"]},{"osd":7,"uuid":"e53ae885-23b6-46fd-b4b1-c62f31a6921f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6817","nonce":820310228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6819","nonce":820310228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6823","nonce":820310228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6821","nonce":820310228}]},"public_addr":"172.21.15.50:6817/820310228","cluster_addr":"172.21.15.50:6819/820310228","heartbeat_back_addr":"172.21.15.50:6823/820310228","heartbeat_front_addr":"172.21.15.50:6821/820310228","state":["exists","up"]},{"osd":8,"uuid":"356e2692-16a9-4045-b58f-c8f9cb5d7241","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6817","nonce":1824918711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6819","nonce":1824918711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6823","nonce":1824918711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6821","nonce":1824918711}]},"public_addr":"172.21.15.142:6817/1824918711","cluster_addr":"172.21.15.142:6819/1824918711","heartbeat_back_addr":"172.21.15.142:6823/1824918711","heartbeat_front_addr":"172.21.15.142:6821/1824918711","state":["exists","up"]},{"osd":9,"uuid":"8e55570e-77e5-4892-8b9f-77dcd03a4eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6827","nonce":3601947736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6829","nonce":3601947736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6833","nonce":3601947736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6831","nonce":3601947736}]},"public_addr":"172.21.15.46:6827/3601947736","cluster_addr":"172.21.15.46:6829/3601947736","heartbeat_back_addr":"172.21.15.46:6833/3601947736","heartbeat_front_addr":"172.21.15.46:6831/3601947736","state":["exists","up"]},{"osd":10,"uuid":"366389fa-fb9a-408b-b5c2-343c054c5a64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6825","nonce":3135765006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6827","nonce":3135765006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6831","nonce":3135765006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6829","nonce":3135765006}]},"public_addr":"172.21.15.50:6825/3135765006","cluster_addr":"172.21.15.50:6827/3135765006","heartbeat_back_addr":"172.21.15.50:6831/3135765006","heartbeat_front_addr":"172.21.15.50:6829/3135765006","state":["exists","up"]},{"osd":11,"uuid":"7b6e3d93-59e1-4853-b4a2-261b728c47a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6825","nonce":1482712612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6827","nonce":1482712612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6831","nonce":1482712612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6829","nonce":1482712612}]},"public_addr":"172.21.15.142:6825/1482712612","cluster_addr":"172.21.15.142:6827/1482712612","heartbeat_back_addr":"172.21.15.142:6831/1482712612","heartbeat_front_addr":"172.21.15.142:6829/1482712612","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:08.082015+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:32.634928+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:55.978864+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:21.796863+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:46.862518+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:11.440432+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:37.764777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:03.374958+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:28.714052+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:55.761167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:22.330951+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:47.839334+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/2465242088":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6801/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6800/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/1381997090":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/3507268217":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2570024356":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/2691061938":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2775954801":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6800/2896098005":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6801/2896098005":"2024-04-27T21:25:02.078730+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:32:52.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.384+0000 7eff5b319640 1 -- 172.21.15.46:0/1925659073 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7eff3c0a3bb0 msgr2=0x7eff3c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:52.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.384+0000 7eff5b319640 1 --2- 172.21.15.46:0/1925659073 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7eff3c0a3bb0 0x7eff3c0a6070 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7eff4c000c00 tx=0x7eff4c0314a0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.384+0000 7eff5b319640 1 -- 172.21.15.46:0/1925659073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff54102560 msgr2=0x7eff541a2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:52.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.384+0000 7eff5b319640 1 --2- 172.21.15.46:0/1925659073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff54102560 0x7eff541a2320 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7eff44009fd0 tx=0x7eff4400b260 comp rx=0 tx=0).stop 2024-04-26T21:32:52.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.384+0000 7eff5b319640 1 -- 172.21.15.46:0/1925659073 shutdown_connections 2024-04-26T21:32:52.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.384+0000 7eff5b319640 1 --2- 172.21.15.46:0/1925659073 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7eff3c0a3bb0 0x7eff3c0a6070 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.384+0000 7eff5b319640 1 --2- 172.21.15.46:0/1925659073 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff54102560 0x7eff541a2320 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.384+0000 7eff5b319640 1 --2- 172.21.15.46:0/1925659073 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7eff54101bc0 0x7eff5419b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.384+0000 7eff5b319640 1 --2- 172.21.15.46:0/1925659073 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7eff54101260 0x7eff5419ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.384+0000 7eff5b319640 1 -- 172.21.15.46:0/1925659073 >> 172.21.15.46:0/1925659073 conn(0x7eff540fcde0 msgr2=0x7eff540fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:52.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.384+0000 7eff5b319640 1 -- 172.21.15.46:0/1925659073 shutdown_connections 2024-04-26T21:32:52.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.384+0000 7eff5b319640 1 -- 172.21.15.46:0/1925659073 wait complete. 2024-04-26T21:32:52.396 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-26T21:32:52.397 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T21:32:52.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.497+0000 7f24c7c93640 1 -- 172.21.15.46:0/2122985033 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24c0103420 msgr2=0x7f24c0109cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:52.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.497+0000 7f24c7c93640 1 --2- 172.21.15.46:0/2122985033 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24c0103420 0x7f24c0109cb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f24b0009ca0 tx=0x7f24b002a8c0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.498+0000 7f24c7c93640 1 -- 172.21.15.46:0/2122985033 shutdown_connections 2024-04-26T21:32:52.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.498+0000 7f24c7c93640 1 --2- 172.21.15.46:0/2122985033 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24c0103420 0x7f24c0109cb0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.498+0000 7f24c7c93640 1 --2- 172.21.15.46:0/2122985033 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24c0102a60 0x7f24c0102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.498+0000 7f24c7c93640 1 --2- 172.21.15.46:0/2122985033 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24c0101870 0x7f24c0101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.498+0000 7f24c7c93640 1 -- 172.21.15.46:0/2122985033 >> 172.21.15.46:0/2122985033 conn(0x7f24c00fcde0 msgr2=0x7f24c00ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:52.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.498+0000 7f24c7c93640 1 -- 172.21.15.46:0/2122985033 shutdown_connections 2024-04-26T21:32:52.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.498+0000 7f24c7c93640 1 -- 172.21.15.46:0/2122985033 wait complete. 2024-04-26T21:32:52.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.498+0000 7f24c7c93640 1 Processor -- start 2024-04-26T21:32:52.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.498+0000 7f24c7c93640 1 -- start start 2024-04-26T21:32:52.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.498+0000 7f24c7c93640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24c0101870 0x7f24c006a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:52.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.498+0000 7f24c7c93640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24c0102a60 0x7f24c019d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:52.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.498+0000 7f24c7c93640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24c0103420 0x7f24c019fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:52.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.498+0000 7f24c7c93640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f24c010d380 con 0x7f24c0101870 2024-04-26T21:32:52.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.498+0000 7f24c7c93640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f24c010d200 con 0x7f24c0102a60 2024-04-26T21:32:52.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.498+0000 7f24c7c93640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f24c010d500 con 0x7f24c0103420 2024-04-26T21:32:52.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.498+0000 7f24c5a2f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24c0101870 0x7f24c006a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:52.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.498+0000 7f24c5a2f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24c0101870 0x7f24c006a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49554/0 (socket says 172.21.15.46:49554) 2024-04-26T21:32:52.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.499+0000 7f24c5a2f640 1 -- 172.21.15.46:0/1258341665 learned_addr learned my addr 172.21.15.46:0/1258341665 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:52.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.499+0000 7f24c522e640 1 --2- 172.21.15.46:0/1258341665 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24c0102a60 0x7f24c019d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:52.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.499+0000 7f24c6230640 1 --2- 172.21.15.46:0/1258341665 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24c0103420 0x7f24c019fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:52.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.499+0000 7f24c5a2f640 1 -- 172.21.15.46:0/1258341665 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24c0103420 msgr2=0x7f24c019fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:52.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.499+0000 7f24c5a2f640 1 --2- 172.21.15.46:0/1258341665 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24c0103420 0x7f24c019fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.499+0000 7f24c5a2f640 1 -- 172.21.15.46:0/1258341665 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24c0102a60 msgr2=0x7f24c019d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:52.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.499+0000 7f24c5a2f640 1 --2- 172.21.15.46:0/1258341665 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24c0102a60 0x7f24c019d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.499+0000 7f24c5a2f640 1 -- 172.21.15.46:0/1258341665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24c01a07a0 con 0x7f24c0101870 2024-04-26T21:32:52.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.499+0000 7f24c6230640 1 --2- 172.21.15.46:0/1258341665 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24c0103420 0x7f24c019fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:32:52.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.499+0000 7f24c5a2f640 1 --2- 172.21.15.46:0/1258341665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24c0101870 0x7f24c006a630 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f24b800e9c0 tx=0x7f24b800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:52.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.499+0000 7f24beffd640 1 -- 172.21.15.46:0/1258341665 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b800cd80 con 0x7f24c0101870 2024-04-26T21:32:52.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.499+0000 7f24c7c93640 1 -- 172.21.15.46:0/1258341665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24c01a0a30 con 0x7f24c0101870 2024-04-26T21:32:52.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.499+0000 7f24beffd640 1 -- 172.21.15.46:0/1258341665 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24b80035f0 con 0x7f24c0101870 2024-04-26T21:32:52.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.499+0000 7f24beffd640 1 -- 172.21.15.46:0/1258341665 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b8010690 con 0x7f24c0101870 2024-04-26T21:32:52.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.499+0000 7f24c7c93640 1 -- 172.21.15.46:0/1258341665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24c01a0fd0 con 0x7f24c0101870 2024-04-26T21:32:52.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.500+0000 7f24c7c93640 1 -- 172.21.15.46:0/1258341665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24c0108c10 con 0x7f24c0101870 2024-04-26T21:32:52.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.501+0000 7f24beffd640 1 -- 172.21.15.46:0/1258341665 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f24b80037a0 con 0x7f24c0101870 2024-04-26T21:32:52.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.502+0000 7f24beffd640 1 --2- 172.21.15.46:0/1258341665 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f24a80a3bb0 0x7f24a80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:52.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.502+0000 7f24c522e640 1 --2- 172.21.15.46:0/1258341665 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f24a80a3bb0 0x7f24a80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:52.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.502+0000 7f24beffd640 1 -- 172.21.15.46:0/1258341665 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f24b80d0760 con 0x7f24c0101870 2024-04-26T21:32:52.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.502+0000 7f24c522e640 1 --2- 172.21.15.46:0/1258341665 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f24a80a3bb0 0x7f24a80a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f24c006a3a0 tx=0x7f24b400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:52.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.506+0000 7f24beffd640 1 -- 172.21.15.46:0/1258341665 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f24b809d1f0 con 0x7f24c0101870 2024-04-26T21:32:52.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.690+0000 7f24c7c93640 1 -- 172.21.15.46:0/1258341665 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f24c0062d70 con 0x7f24a80a3bb0 2024-04-26T21:32:52.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.698+0000 7f24beffd640 1 -- 172.21.15.46:0/1258341665 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258901 (secure 0 0 0) 0x7f24c0062d70 con 0x7f24a80a3bb0 2024-04-26T21:32:52.703 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:32:52.705 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-04-26T21:32:52.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.702+0000 7f24c7c93640 1 -- 172.21.15.46:0/1258341665 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f24a80a3bb0 msgr2=0x7f24a80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:52.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.702+0000 7f24c7c93640 1 --2- 172.21.15.46:0/1258341665 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f24a80a3bb0 0x7f24a80a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f24c006a3a0 tx=0x7f24b400b410 comp rx=0 tx=0).stop 2024-04-26T21:32:52.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.702+0000 7f24c7c93640 1 -- 172.21.15.46:0/1258341665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24c0101870 msgr2=0x7f24c006a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:52.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.702+0000 7f24c7c93640 1 --2- 172.21.15.46:0/1258341665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24c0101870 0x7f24c006a630 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f24b800e9c0 tx=0x7f24b800ee80 comp rx=0 tx=0).stop 2024-04-26T21:32:52.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.702+0000 7f24c7c93640 1 -- 172.21.15.46:0/1258341665 shutdown_connections 2024-04-26T21:32:52.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.702+0000 7f24c7c93640 1 --2- 172.21.15.46:0/1258341665 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f24a80a3bb0 0x7f24a80a6070 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.702+0000 7f24c7c93640 1 --2- 172.21.15.46:0/1258341665 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24c0103420 0x7f24c019fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.702+0000 7f24c7c93640 1 --2- 172.21.15.46:0/1258341665 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24c0102a60 0x7f24c019d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.702+0000 7f24c7c93640 1 --2- 172.21.15.46:0/1258341665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24c0101870 0x7f24c006a630 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.702+0000 7f24c7c93640 1 -- 172.21.15.46:0/1258341665 >> 172.21.15.46:0/1258341665 conn(0x7f24c00fcde0 msgr2=0x7f24c0105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:52.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.702+0000 7f24c7c93640 1 -- 172.21.15.46:0/1258341665 shutdown_connections 2024-04-26T21:32:52.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.702+0000 7f24c7c93640 1 -- 172.21.15.46:0/1258341665 wait complete. 2024-04-26T21:32:52.714 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-04-26T21:32:51.931078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":131,"num_read_kb":114,"num_write":286,"num_write_kb":2975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":381020,"kb_used_data":11044,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124560804,"statfs":{"total":1151940427776,"available":1151550263296,"internally_reserved":0,"allocated":11309056,"data_stored":7048072,"data_compressed":107181,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19140,"internal_metadata":378713404},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4640,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":105,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001775"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905654+0000","last_change":"2024-04-26T21:32:44.863556+0000","last_active":"2024-04-26T21:32:45.905654+0000","last_peered":"2024-04-26T21:32:45.905654+0000","last_clean":"2024-04-26T21:32:45.905654+0000","last_became_active":"2024-04-26T21:32:44.863217+0000","last_became_peered":"2024-04-26T21:32:44.863217+0000","last_unstale":"2024-04-26T21:32:45.905654+0000","last_undegraded":"2024-04-26T21:32:45.905654+0000","last_fullsized":"2024-04-26T21:32:45.905654+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:36:19.416660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904461+0000","last_change":"2024-04-26T21:32:44.862340+0000","last_active":"2024-04-26T21:32:45.904461+0000","last_peered":"2024-04-26T21:32:45.904461+0000","last_clean":"2024-04-26T21:32:45.904461+0000","last_became_active":"2024-04-26T21:32:44.861753+0000","last_became_peered":"2024-04-26T21:32:44.861753+0000","last_unstale":"2024-04-26T21:32:45.904461+0000","last_undegraded":"2024-04-26T21:32:45.904461+0000","last_fullsized":"2024-04-26T21:32:45.904461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:56:46.395806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904731+0000","last_change":"2024-04-26T21:32:44.865678+0000","last_active":"2024-04-26T21:32:45.904731+0000","last_peered":"2024-04-26T21:32:45.904731+0000","last_clean":"2024-04-26T21:32:45.904731+0000","last_became_active":"2024-04-26T21:32:44.865001+0000","last_became_peered":"2024-04-26T21:32:44.865001+0000","last_unstale":"2024-04-26T21:32:45.904731+0000","last_undegraded":"2024-04-26T21:32:45.904731+0000","last_fullsized":"2024-04-26T21:32:45.904731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:29:42.803401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905389+0000","last_change":"2024-04-26T21:32:44.869469+0000","last_active":"2024-04-26T21:32:45.905389+0000","last_peered":"2024-04-26T21:32:45.905389+0000","last_clean":"2024-04-26T21:32:45.905389+0000","last_became_active":"2024-04-26T21:32:44.868443+0000","last_became_peered":"2024-04-26T21:32:44.868443+0000","last_unstale":"2024-04-26T21:32:45.905389+0000","last_undegraded":"2024-04-26T21:32:45.905389+0000","last_fullsized":"2024-04-26T21:32:45.905389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:41:43.252283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905138+0000","last_change":"2024-04-26T21:32:44.855500+0000","last_active":"2024-04-26T21:32:45.905138+0000","last_peered":"2024-04-26T21:32:45.905138+0000","last_clean":"2024-04-26T21:32:45.905138+0000","last_became_active":"2024-04-26T21:32:44.854902+0000","last_became_peered":"2024-04-26T21:32:44.854902+0000","last_unstale":"2024-04-26T21:32:45.905138+0000","last_undegraded":"2024-04-26T21:32:45.905138+0000","last_fullsized":"2024-04-26T21:32:45.905138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:14:02.135590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.851865+0000","last_change":"2024-04-26T21:32:44.864742+0000","last_active":"2024-04-26T21:32:45.851865+0000","last_peered":"2024-04-26T21:32:45.851865+0000","last_clean":"2024-04-26T21:32:45.851865+0000","last_became_active":"2024-04-26T21:32:44.864369+0000","last_became_peered":"2024-04-26T21:32:44.864369+0000","last_unstale":"2024-04-26T21:32:45.851865+0000","last_undegraded":"2024-04-26T21:32:45.851865+0000","last_fullsized":"2024-04-26T21:32:45.851865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:24:43.757233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:46.094853+0000","last_change":"2024-04-26T21:32:44.863158+0000","last_active":"2024-04-26T21:32:46.094853+0000","last_peered":"2024-04-26T21:32:46.094853+0000","last_clean":"2024-04-26T21:32:46.094853+0000","last_became_active":"2024-04-26T21:32:44.862773+0000","last_became_peered":"2024-04-26T21:32:44.862773+0000","last_unstale":"2024-04-26T21:32:46.094853+0000","last_undegraded":"2024-04-26T21:32:46.094853+0000","last_fullsized":"2024-04-26T21:32:46.094853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:15:44.907270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904664+0000","last_change":"2024-04-26T21:32:44.864585+0000","last_active":"2024-04-26T21:32:45.904664+0000","last_peered":"2024-04-26T21:32:45.904664+0000","last_clean":"2024-04-26T21:32:45.904664+0000","last_became_active":"2024-04-26T21:32:44.863572+0000","last_became_peered":"2024-04-26T21:32:44.863572+0000","last_unstale":"2024-04-26T21:32:45.904664+0000","last_undegraded":"2024-04-26T21:32:45.904664+0000","last_fullsized":"2024-04-26T21:32:45.904664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:32:42.481698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905268+0000","last_change":"2024-04-26T21:32:44.855339+0000","last_active":"2024-04-26T21:32:45.905268+0000","last_peered":"2024-04-26T21:32:45.905268+0000","last_clean":"2024-04-26T21:32:45.905268+0000","last_became_active":"2024-04-26T21:32:44.853923+0000","last_became_peered":"2024-04-26T21:32:44.853923+0000","last_unstale":"2024-04-26T21:32:45.905268+0000","last_undegraded":"2024-04-26T21:32:45.905268+0000","last_fullsized":"2024-04-26T21:32:45.905268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:08:03.274120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.851852+0000","last_change":"2024-04-26T21:32:44.865564+0000","last_active":"2024-04-26T21:32:45.851852+0000","last_peered":"2024-04-26T21:32:45.851852+0000","last_clean":"2024-04-26T21:32:45.851852+0000","last_became_active":"2024-04-26T21:32:44.865209+0000","last_became_peered":"2024-04-26T21:32:44.865209+0000","last_unstale":"2024-04-26T21:32:45.851852+0000","last_undegraded":"2024-04-26T21:32:45.851852+0000","last_fullsized":"2024-04-26T21:32:45.851852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:51:36.388624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:46.095218+0000","last_change":"2024-04-26T21:32:44.865437+0000","last_active":"2024-04-26T21:32:46.095218+0000","last_peered":"2024-04-26T21:32:46.095218+0000","last_clean":"2024-04-26T21:32:46.095218+0000","last_became_active":"2024-04-26T21:32:44.864662+0000","last_became_peered":"2024-04-26T21:32:44.864662+0000","last_unstale":"2024-04-26T21:32:46.095218+0000","last_undegraded":"2024-04-26T21:32:46.095218+0000","last_fullsized":"2024-04-26T21:32:46.095218+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:23:37.867338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905241+0000","last_change":"2024-04-26T21:32:44.869738+0000","last_active":"2024-04-26T21:32:45.905241+0000","last_peered":"2024-04-26T21:32:45.905241+0000","last_clean":"2024-04-26T21:32:45.905241+0000","last_became_active":"2024-04-26T21:32:44.869133+0000","last_became_peered":"2024-04-26T21:32:44.869133+0000","last_unstale":"2024-04-26T21:32:45.905241+0000","last_undegraded":"2024-04-26T21:32:45.905241+0000","last_fullsized":"2024-04-26T21:32:45.905241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:08:29.233128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.850576+0000","last_change":"2024-04-26T21:32:44.856970+0000","last_active":"2024-04-26T21:32:45.850576+0000","last_peered":"2024-04-26T21:32:45.850576+0000","last_clean":"2024-04-26T21:32:45.850576+0000","last_became_active":"2024-04-26T21:32:44.856579+0000","last_became_peered":"2024-04-26T21:32:44.856579+0000","last_unstale":"2024-04-26T21:32:45.850576+0000","last_undegraded":"2024-04-26T21:32:45.850576+0000","last_fullsized":"2024-04-26T21:32:45.850576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:24:17.281094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904282+0000","last_change":"2024-04-26T21:32:44.859688+0000","last_active":"2024-04-26T21:32:45.904282+0000","last_peered":"2024-04-26T21:32:45.904282+0000","last_clean":"2024-04-26T21:32:45.904282+0000","last_became_active":"2024-04-26T21:32:44.859336+0000","last_became_peered":"2024-04-26T21:32:44.859336+0000","last_unstale":"2024-04-26T21:32:45.904282+0000","last_undegraded":"2024-04-26T21:32:45.904282+0000","last_fullsized":"2024-04-26T21:32:45.904282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:55:46.956334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904017+0000","last_change":"2024-04-26T21:32:44.860647+0000","last_active":"2024-04-26T21:32:45.904017+0000","last_peered":"2024-04-26T21:32:45.904017+0000","last_clean":"2024-04-26T21:32:45.904017+0000","last_became_active":"2024-04-26T21:32:44.860237+0000","last_became_peered":"2024-04-26T21:32:44.860237+0000","last_unstale":"2024-04-26T21:32:45.904017+0000","last_undegraded":"2024-04-26T21:32:45.904017+0000","last_fullsized":"2024-04-26T21:32:45.904017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:49:10.042990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905459+0000","last_change":"2024-04-26T21:32:44.865289+0000","last_active":"2024-04-26T21:32:45.905459+0000","last_peered":"2024-04-26T21:32:45.905459+0000","last_clean":"2024-04-26T21:32:45.905459+0000","last_became_active":"2024-04-26T21:32:44.864951+0000","last_became_peered":"2024-04-26T21:32:44.864951+0000","last_unstale":"2024-04-26T21:32:45.905459+0000","last_undegraded":"2024-04-26T21:32:45.905459+0000","last_fullsized":"2024-04-26T21:32:45.905459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:31:51.998634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905976+0000","last_change":"2024-04-26T21:32:44.857616+0000","last_active":"2024-04-26T21:32:45.905976+0000","last_peered":"2024-04-26T21:32:45.905976+0000","last_clean":"2024-04-26T21:32:45.905976+0000","last_became_active":"2024-04-26T21:32:44.857259+0000","last_became_peered":"2024-04-26T21:32:44.857259+0000","last_unstale":"2024-04-26T21:32:45.905976+0000","last_undegraded":"2024-04-26T21:32:45.905976+0000","last_fullsized":"2024-04-26T21:32:45.905976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:59:37.412657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905236+0000","last_change":"2024-04-26T21:32:44.857177+0000","last_active":"2024-04-26T21:32:45.905236+0000","last_peered":"2024-04-26T21:32:45.905236+0000","last_clean":"2024-04-26T21:32:45.905236+0000","last_became_active":"2024-04-26T21:32:44.856810+0000","last_became_peered":"2024-04-26T21:32:44.856810+0000","last_unstale":"2024-04-26T21:32:45.905236+0000","last_undegraded":"2024-04-26T21:32:45.905236+0000","last_fullsized":"2024-04-26T21:32:45.905236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:31:40.712896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.851995+0000","last_change":"2024-04-26T21:32:44.870498+0000","last_active":"2024-04-26T21:32:45.851995+0000","last_peered":"2024-04-26T21:32:45.851995+0000","last_clean":"2024-04-26T21:32:45.851995+0000","last_became_active":"2024-04-26T21:32:44.870170+0000","last_became_peered":"2024-04-26T21:32:44.870170+0000","last_unstale":"2024-04-26T21:32:45.851995+0000","last_undegraded":"2024-04-26T21:32:45.851995+0000","last_fullsized":"2024-04-26T21:32:45.851995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:27.428027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905364+0000","last_change":"2024-04-26T21:32:43.845192+0000","last_active":"2024-04-26T21:32:45.905364+0000","last_peered":"2024-04-26T21:32:45.905364+0000","last_clean":"2024-04-26T21:32:45.905364+0000","last_became_active":"2024-04-26T21:32:43.844862+0000","last_became_peered":"2024-04-26T21:32:43.844862+0000","last_unstale":"2024-04-26T21:32:45.905364+0000","last_undegraded":"2024-04-26T21:32:45.905364+0000","last_fullsized":"2024-04-26T21:32:45.905364+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:07:00.060047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905637+0000","last_change":"2024-04-26T21:32:44.867358+0000","last_active":"2024-04-26T21:32:45.905637+0000","last_peered":"2024-04-26T21:32:45.905637+0000","last_clean":"2024-04-26T21:32:45.905637+0000","last_became_active":"2024-04-26T21:32:44.864833+0000","last_became_peered":"2024-04-26T21:32:44.864833+0000","last_unstale":"2024-04-26T21:32:45.905637+0000","last_undegraded":"2024-04-26T21:32:45.905637+0000","last_fullsized":"2024-04-26T21:32:45.905637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:45:50.285471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905648+0000","last_change":"2024-04-26T21:32:43.849724+0000","last_active":"2024-04-26T21:32:45.905648+0000","last_peered":"2024-04-26T21:32:45.905648+0000","last_clean":"2024-04-26T21:32:45.905648+0000","last_became_active":"2024-04-26T21:32:43.849040+0000","last_became_peered":"2024-04-26T21:32:43.849040+0000","last_unstale":"2024-04-26T21:32:45.905648+0000","last_undegraded":"2024-04-26T21:32:45.905648+0000","last_fullsized":"2024-04-26T21:32:45.905648+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:48.994093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904631+0000","last_change":"2024-04-26T21:32:44.863095+0000","last_active":"2024-04-26T21:32:45.904631+0000","last_peered":"2024-04-26T21:32:45.904631+0000","last_clean":"2024-04-26T21:32:45.904631+0000","last_became_active":"2024-04-26T21:32:44.862739+0000","last_became_peered":"2024-04-26T21:32:44.862739+0000","last_unstale":"2024-04-26T21:32:45.904631+0000","last_undegraded":"2024-04-26T21:32:45.904631+0000","last_fullsized":"2024-04-26T21:32:45.904631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:21:19.927248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:48.894281+0000","last_change":"2024-04-26T21:32:43.854315+0000","last_active":"2024-04-26T21:32:48.894281+0000","last_peered":"2024-04-26T21:32:48.894281+0000","last_clean":"2024-04-26T21:32:48.894281+0000","last_became_active":"2024-04-26T21:32:43.849472+0000","last_became_peered":"2024-04-26T21:32:43.849472+0000","last_unstale":"2024-04-26T21:32:48.894281+0000","last_undegraded":"2024-04-26T21:32:48.894281+0000","last_fullsized":"2024-04-26T21:32:48.894281+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:42:24.213303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904901+0000","last_change":"2024-04-26T21:32:44.860100+0000","last_active":"2024-04-26T21:32:45.904901+0000","last_peered":"2024-04-26T21:32:45.904901+0000","last_clean":"2024-04-26T21:32:45.904901+0000","last_became_active":"2024-04-26T21:32:44.858890+0000","last_became_peered":"2024-04-26T21:32:44.858890+0000","last_unstale":"2024-04-26T21:32:45.904901+0000","last_undegraded":"2024-04-26T21:32:45.904901+0000","last_fullsized":"2024-04-26T21:32:45.904901+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:22.749764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905078+0000","last_change":"2024-04-26T21:32:43.840719+0000","last_active":"2024-04-26T21:32:45.905078+0000","last_peered":"2024-04-26T21:32:45.905078+0000","last_clean":"2024-04-26T21:32:45.905078+0000","last_became_active":"2024-04-26T21:32:43.840205+0000","last_became_peered":"2024-04-26T21:32:43.840205+0000","last_unstale":"2024-04-26T21:32:45.905078+0000","last_undegraded":"2024-04-26T21:32:45.905078+0000","last_fullsized":"2024-04-26T21:32:45.905078+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:29:41.025573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905966+0000","last_change":"2024-04-26T21:32:44.858287+0000","last_active":"2024-04-26T21:32:45.905966+0000","last_peered":"2024-04-26T21:32:45.905966+0000","last_clean":"2024-04-26T21:32:45.905966+0000","last_became_active":"2024-04-26T21:32:44.857890+0000","last_became_peered":"2024-04-26T21:32:44.857890+0000","last_unstale":"2024-04-26T21:32:45.905966+0000","last_undegraded":"2024-04-26T21:32:45.905966+0000","last_fullsized":"2024-04-26T21:32:45.905966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:16:23.743633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:47.886087+0000","last_change":"2024-04-26T21:32:43.850841+0000","last_active":"2024-04-26T21:32:47.886087+0000","last_peered":"2024-04-26T21:32:47.886087+0000","last_clean":"2024-04-26T21:32:47.886087+0000","last_became_active":"2024-04-26T21:32:43.850512+0000","last_became_peered":"2024-04-26T21:32:43.850512+0000","last_unstale":"2024-04-26T21:32:47.886087+0000","last_undegraded":"2024-04-26T21:32:47.886087+0000","last_fullsized":"2024-04-26T21:32:47.886087+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:13:51.814145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905362+0000","last_change":"2024-04-26T21:32:44.853438+0000","last_active":"2024-04-26T21:32:45.905362+0000","last_peered":"2024-04-26T21:32:45.905362+0000","last_clean":"2024-04-26T21:32:45.905362+0000","last_became_active":"2024-04-26T21:32:44.853126+0000","last_became_peered":"2024-04-26T21:32:44.853126+0000","last_unstale":"2024-04-26T21:32:45.905362+0000","last_undegraded":"2024-04-26T21:32:45.905362+0000","last_fullsized":"2024-04-26T21:32:45.905362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:43.054626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:48.895433+0000","last_change":"2024-04-26T21:32:43.839230+0000","last_active":"2024-04-26T21:32:48.895433+0000","last_peered":"2024-04-26T21:32:48.895433+0000","last_clean":"2024-04-26T21:32:48.895433+0000","last_became_active":"2024-04-26T21:32:43.838894+0000","last_became_peered":"2024-04-26T21:32:43.838894+0000","last_unstale":"2024-04-26T21:32:48.895433+0000","last_undegraded":"2024-04-26T21:32:48.895433+0000","last_fullsized":"2024-04-26T21:32:48.895433+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:54:29.718314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905045+0000","last_change":"2024-04-26T21:32:44.859870+0000","last_active":"2024-04-26T21:32:45.905045+0000","last_peered":"2024-04-26T21:32:45.905045+0000","last_clean":"2024-04-26T21:32:45.905045+0000","last_became_active":"2024-04-26T21:32:44.859474+0000","last_became_peered":"2024-04-26T21:32:44.859474+0000","last_unstale":"2024-04-26T21:32:45.905045+0000","last_undegraded":"2024-04-26T21:32:45.905045+0000","last_fullsized":"2024-04-26T21:32:45.905045+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:13:14.854523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:48.893992+0000","last_change":"2024-04-26T21:32:43.841090+0000","last_active":"2024-04-26T21:32:48.893992+0000","last_peered":"2024-04-26T21:32:48.893992+0000","last_clean":"2024-04-26T21:32:48.893992+0000","last_became_active":"2024-04-26T21:32:43.840723+0000","last_became_peered":"2024-04-26T21:32:43.840723+0000","last_unstale":"2024-04-26T21:32:48.893992+0000","last_undegraded":"2024-04-26T21:32:48.893992+0000","last_fullsized":"2024-04-26T21:32:48.893992+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:36:45.799738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904881+0000","last_change":"2024-04-26T21:32:44.869439+0000","last_active":"2024-04-26T21:32:45.904881+0000","last_peered":"2024-04-26T21:32:45.904881+0000","last_clean":"2024-04-26T21:32:45.904881+0000","last_became_active":"2024-04-26T21:32:44.868696+0000","last_became_peered":"2024-04-26T21:32:44.868696+0000","last_unstale":"2024-04-26T21:32:45.904881+0000","last_undegraded":"2024-04-26T21:32:45.904881+0000","last_fullsized":"2024-04-26T21:32:45.904881+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:46.400937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905617+0000","last_change":"2024-04-26T21:32:43.852431+0000","last_active":"2024-04-26T21:32:45.905617+0000","last_peered":"2024-04-26T21:32:45.905617+0000","last_clean":"2024-04-26T21:32:45.905617+0000","last_became_active":"2024-04-26T21:32:43.852143+0000","last_became_peered":"2024-04-26T21:32:43.852143+0000","last_unstale":"2024-04-26T21:32:45.905617+0000","last_undegraded":"2024-04-26T21:32:45.905617+0000","last_fullsized":"2024-04-26T21:32:45.905617+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:35.971582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904878+0000","last_change":"2024-04-26T21:32:44.861118+0000","last_active":"2024-04-26T21:32:45.904878+0000","last_peered":"2024-04-26T21:32:45.904878+0000","last_clean":"2024-04-26T21:32:45.904878+0000","last_became_active":"2024-04-26T21:32:44.860765+0000","last_became_peered":"2024-04-26T21:32:44.860765+0000","last_unstale":"2024-04-26T21:32:45.904878+0000","last_undegraded":"2024-04-26T21:32:45.904878+0000","last_fullsized":"2024-04-26T21:32:45.904878+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:01:09.370401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:47.885238+0000","last_change":"2024-04-26T21:32:43.840267+0000","last_active":"2024-04-26T21:32:47.885238+0000","last_peered":"2024-04-26T21:32:47.885238+0000","last_clean":"2024-04-26T21:32:47.885238+0000","last_became_active":"2024-04-26T21:32:43.839944+0000","last_became_peered":"2024-04-26T21:32:43.839944+0000","last_unstale":"2024-04-26T21:32:47.885238+0000","last_undegraded":"2024-04-26T21:32:47.885238+0000","last_fullsized":"2024-04-26T21:32:47.885238+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:46:13.671063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905332+0000","last_change":"2024-04-26T21:32:44.861772+0000","last_active":"2024-04-26T21:32:45.905332+0000","last_peered":"2024-04-26T21:32:45.905332+0000","last_clean":"2024-04-26T21:32:45.905332+0000","last_became_active":"2024-04-26T21:32:44.860376+0000","last_became_peered":"2024-04-26T21:32:44.860376+0000","last_unstale":"2024-04-26T21:32:45.905332+0000","last_undegraded":"2024-04-26T21:32:45.905332+0000","last_fullsized":"2024-04-26T21:32:45.905332+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:07:29.499706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:47.885591+0000","last_change":"2024-04-26T21:32:43.849923+0000","last_active":"2024-04-26T21:32:47.885591+0000","last_peered":"2024-04-26T21:32:47.885591+0000","last_clean":"2024-04-26T21:32:47.885591+0000","last_became_active":"2024-04-26T21:32:43.848193+0000","last_became_peered":"2024-04-26T21:32:43.848193+0000","last_unstale":"2024-04-26T21:32:47.885591+0000","last_undegraded":"2024-04-26T21:32:47.885591+0000","last_fullsized":"2024-04-26T21:32:47.885591+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:06:00.247084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904552+0000","last_change":"2024-04-26T21:32:44.866684+0000","last_active":"2024-04-26T21:32:45.904552+0000","last_peered":"2024-04-26T21:32:45.904552+0000","last_clean":"2024-04-26T21:32:45.904552+0000","last_became_active":"2024-04-26T21:32:44.866350+0000","last_became_peered":"2024-04-26T21:32:44.866350+0000","last_unstale":"2024-04-26T21:32:45.904552+0000","last_undegraded":"2024-04-26T21:32:45.904552+0000","last_fullsized":"2024-04-26T21:32:45.904552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:21.306500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.906114+0000","last_change":"2024-04-26T21:32:43.841617+0000","last_active":"2024-04-26T21:32:45.906114+0000","last_peered":"2024-04-26T21:32:45.906114+0000","last_clean":"2024-04-26T21:32:45.906114+0000","last_became_active":"2024-04-26T21:32:43.841302+0000","last_became_peered":"2024-04-26T21:32:43.841302+0000","last_unstale":"2024-04-26T21:32:45.906114+0000","last_undegraded":"2024-04-26T21:32:45.906114+0000","last_fullsized":"2024-04-26T21:32:45.906114+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:16:25.409255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905147+0000","last_change":"2024-04-26T21:32:44.860793+0000","last_active":"2024-04-26T21:32:45.905147+0000","last_peered":"2024-04-26T21:32:45.905147+0000","last_clean":"2024-04-26T21:32:45.905147+0000","last_became_active":"2024-04-26T21:32:44.859418+0000","last_became_peered":"2024-04-26T21:32:44.859418+0000","last_unstale":"2024-04-26T21:32:45.905147+0000","last_undegraded":"2024-04-26T21:32:45.905147+0000","last_fullsized":"2024-04-26T21:32:45.905147+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:17:46.189899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'8","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:47.884595+0000","last_change":"2024-04-26T21:32:43.843122+0000","last_active":"2024-04-26T21:32:47.884595+0000","last_peered":"2024-04-26T21:32:47.884595+0000","last_clean":"2024-04-26T21:32:47.884595+0000","last_became_active":"2024-04-26T21:32:43.842751+0000","last_became_peered":"2024-04-26T21:32:43.842751+0000","last_unstale":"2024-04-26T21:32:47.884595+0000","last_undegraded":"2024-04-26T21:32:47.884595+0000","last_fullsized":"2024-04-26T21:32:47.884595+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:16:58.533030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905144+0000","last_change":"2024-04-26T21:32:44.872829+0000","last_active":"2024-04-26T21:32:45.905144+0000","last_peered":"2024-04-26T21:32:45.905144+0000","last_clean":"2024-04-26T21:32:45.905144+0000","last_became_active":"2024-04-26T21:32:44.872466+0000","last_became_peered":"2024-04-26T21:32:44.872466+0000","last_unstale":"2024-04-26T21:32:45.905144+0000","last_undegraded":"2024-04-26T21:32:45.905144+0000","last_fullsized":"2024-04-26T21:32:45.905144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:07:27.698789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.851758+0000","last_change":"2024-04-26T21:32:43.840277+0000","last_active":"2024-04-26T21:32:45.851758+0000","last_peered":"2024-04-26T21:32:45.851758+0000","last_clean":"2024-04-26T21:32:45.851758+0000","last_became_active":"2024-04-26T21:32:43.839962+0000","last_became_peered":"2024-04-26T21:32:43.839962+0000","last_unstale":"2024-04-26T21:32:45.851758+0000","last_undegraded":"2024-04-26T21:32:45.851758+0000","last_fullsized":"2024-04-26T21:32:45.851758+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:25:54.475216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.850259+0000","last_change":"2024-04-26T21:32:44.866465+0000","last_active":"2024-04-26T21:32:45.850259+0000","last_peered":"2024-04-26T21:32:45.850259+0000","last_clean":"2024-04-26T21:32:45.850259+0000","last_became_active":"2024-04-26T21:32:44.866067+0000","last_became_peered":"2024-04-26T21:32:44.866067+0000","last_unstale":"2024-04-26T21:32:45.850259+0000","last_undegraded":"2024-04-26T21:32:45.850259+0000","last_fullsized":"2024-04-26T21:32:45.850259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:47:03.884436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:47.886078+0000","last_change":"2024-04-26T21:32:43.851535+0000","last_active":"2024-04-26T21:32:47.886078+0000","last_peered":"2024-04-26T21:32:47.886078+0000","last_clean":"2024-04-26T21:32:47.886078+0000","last_became_active":"2024-04-26T21:32:43.851237+0000","last_became_peered":"2024-04-26T21:32:43.851237+0000","last_unstale":"2024-04-26T21:32:47.886078+0000","last_undegraded":"2024-04-26T21:32:47.886078+0000","last_fullsized":"2024-04-26T21:32:47.886078+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:08:40.624027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:46.095216+0000","last_change":"2024-04-26T21:32:44.874686+0000","last_active":"2024-04-26T21:32:46.095216+0000","last_peered":"2024-04-26T21:32:46.095216+0000","last_clean":"2024-04-26T21:32:46.095216+0000","last_became_active":"2024-04-26T21:32:44.874377+0000","last_became_peered":"2024-04-26T21:32:44.874377+0000","last_unstale":"2024-04-26T21:32:46.095216+0000","last_undegraded":"2024-04-26T21:32:46.095216+0000","last_fullsized":"2024-04-26T21:32:46.095216+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:45:33.627663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:48.894431+0000","last_change":"2024-04-26T21:32:43.846522+0000","last_active":"2024-04-26T21:32:48.894431+0000","last_peered":"2024-04-26T21:32:48.894431+0000","last_clean":"2024-04-26T21:32:48.894431+0000","last_became_active":"2024-04-26T21:32:43.846138+0000","last_became_peered":"2024-04-26T21:32:43.846138+0000","last_unstale":"2024-04-26T21:32:48.894431+0000","last_undegraded":"2024-04-26T21:32:48.894431+0000","last_fullsized":"2024-04-26T21:32:48.894431+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:28:40.257244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:46.094810+0000","last_change":"2024-04-26T21:32:44.872228+0000","last_active":"2024-04-26T21:32:46.094810+0000","last_peered":"2024-04-26T21:32:46.094810+0000","last_clean":"2024-04-26T21:32:46.094810+0000","last_became_active":"2024-04-26T21:32:44.871818+0000","last_became_peered":"2024-04-26T21:32:44.871818+0000","last_unstale":"2024-04-26T21:32:46.094810+0000","last_undegraded":"2024-04-26T21:32:46.094810+0000","last_fullsized":"2024-04-26T21:32:46.094810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:52:32.520802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:48.894849+0000","last_change":"2024-04-26T21:32:43.846756+0000","last_active":"2024-04-26T21:32:48.894849+0000","last_peered":"2024-04-26T21:32:48.894849+0000","last_clean":"2024-04-26T21:32:48.894849+0000","last_became_active":"2024-04-26T21:32:43.846477+0000","last_became_peered":"2024-04-26T21:32:43.846477+0000","last_unstale":"2024-04-26T21:32:48.894849+0000","last_undegraded":"2024-04-26T21:32:48.894849+0000","last_fullsized":"2024-04-26T21:32:48.894849+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:33:27.859844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904693+0000","last_change":"2024-04-26T21:32:44.859853+0000","last_active":"2024-04-26T21:32:45.904693+0000","last_peered":"2024-04-26T21:32:45.904693+0000","last_clean":"2024-04-26T21:32:45.904693+0000","last_became_active":"2024-04-26T21:32:44.859521+0000","last_became_peered":"2024-04-26T21:32:44.859521+0000","last_unstale":"2024-04-26T21:32:45.904693+0000","last_undegraded":"2024-04-26T21:32:45.904693+0000","last_fullsized":"2024-04-26T21:32:45.904693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:48:54.955913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:49.889630+0000","last_change":"2024-04-26T21:32:43.840795+0000","last_active":"2024-04-26T21:32:49.889630+0000","last_peered":"2024-04-26T21:32:49.889630+0000","last_clean":"2024-04-26T21:32:49.889630+0000","last_became_active":"2024-04-26T21:32:43.840443+0000","last_became_peered":"2024-04-26T21:32:43.840443+0000","last_unstale":"2024-04-26T21:32:49.889630+0000","last_undegraded":"2024-04-26T21:32:49.889630+0000","last_fullsized":"2024-04-26T21:32:49.889630+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:09:48.944085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905287+0000","last_change":"2024-04-26T21:32:44.862344+0000","last_active":"2024-04-26T21:32:45.905287+0000","last_peered":"2024-04-26T21:32:45.905287+0000","last_clean":"2024-04-26T21:32:45.905287+0000","last_became_active":"2024-04-26T21:32:44.862029+0000","last_became_peered":"2024-04-26T21:32:44.862029+0000","last_unstale":"2024-04-26T21:32:45.905287+0000","last_undegraded":"2024-04-26T21:32:45.905287+0000","last_fullsized":"2024-04-26T21:32:45.905287+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:45:48.833753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904823+0000","last_change":"2024-04-26T21:32:43.843853+0000","last_active":"2024-04-26T21:32:45.904823+0000","last_peered":"2024-04-26T21:32:45.904823+0000","last_clean":"2024-04-26T21:32:45.904823+0000","last_became_active":"2024-04-26T21:32:43.843525+0000","last_became_peered":"2024-04-26T21:32:43.843525+0000","last_unstale":"2024-04-26T21:32:45.904823+0000","last_undegraded":"2024-04-26T21:32:45.904823+0000","last_fullsized":"2024-04-26T21:32:45.904823+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:11:43.222849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.851833+0000","last_change":"2024-04-26T21:32:44.869123+0000","last_active":"2024-04-26T21:32:45.851833+0000","last_peered":"2024-04-26T21:32:45.851833+0000","last_clean":"2024-04-26T21:32:45.851833+0000","last_became_active":"2024-04-26T21:32:44.868573+0000","last_became_peered":"2024-04-26T21:32:44.868573+0000","last_unstale":"2024-04-26T21:32:45.851833+0000","last_undegraded":"2024-04-26T21:32:45.851833+0000","last_fullsized":"2024-04-26T21:32:45.851833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:48:54.447157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:46.095005+0000","last_change":"2024-04-26T21:32:43.850596+0000","last_active":"2024-04-26T21:32:46.095005+0000","last_peered":"2024-04-26T21:32:46.095005+0000","last_clean":"2024-04-26T21:32:46.095005+0000","last_became_active":"2024-04-26T21:32:43.850218+0000","last_became_peered":"2024-04-26T21:32:43.850218+0000","last_unstale":"2024-04-26T21:32:46.095005+0000","last_undegraded":"2024-04-26T21:32:46.095005+0000","last_fullsized":"2024-04-26T21:32:46.095005+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:34.068783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.851635+0000","last_change":"2024-04-26T21:32:44.863450+0000","last_active":"2024-04-26T21:32:45.851635+0000","last_peered":"2024-04-26T21:32:45.851635+0000","last_clean":"2024-04-26T21:32:45.851635+0000","last_became_active":"2024-04-26T21:32:44.863089+0000","last_became_peered":"2024-04-26T21:32:44.863089+0000","last_unstale":"2024-04-26T21:32:45.851635+0000","last_undegraded":"2024-04-26T21:32:45.851635+0000","last_fullsized":"2024-04-26T21:32:45.851635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:04:51.821073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905893+0000","last_change":"2024-04-26T21:32:44.855575+0000","last_active":"2024-04-26T21:32:45.905893+0000","last_peered":"2024-04-26T21:32:45.905893+0000","last_clean":"2024-04-26T21:32:45.905893+0000","last_became_active":"2024-04-26T21:32:44.855258+0000","last_became_peered":"2024-04-26T21:32:44.855258+0000","last_unstale":"2024-04-26T21:32:45.905893+0000","last_undegraded":"2024-04-26T21:32:45.905893+0000","last_fullsized":"2024-04-26T21:32:45.905893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:41:07.559095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:48.894928+0000","last_change":"2024-04-26T21:32:43.850713+0000","last_active":"2024-04-26T21:32:48.894928+0000","last_peered":"2024-04-26T21:32:48.894928+0000","last_clean":"2024-04-26T21:32:48.894928+0000","last_became_active":"2024-04-26T21:32:43.850406+0000","last_became_peered":"2024-04-26T21:32:43.850406+0000","last_unstale":"2024-04-26T21:32:48.894928+0000","last_undegraded":"2024-04-26T21:32:48.894928+0000","last_fullsized":"2024-04-26T21:32:48.894928+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:11:02.682963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905520+0000","last_change":"2024-04-26T21:32:44.868317+0000","last_active":"2024-04-26T21:32:45.905520+0000","last_peered":"2024-04-26T21:32:45.905520+0000","last_clean":"2024-04-26T21:32:45.905520+0000","last_became_active":"2024-04-26T21:32:44.864571+0000","last_became_peered":"2024-04-26T21:32:44.864571+0000","last_unstale":"2024-04-26T21:32:45.905520+0000","last_undegraded":"2024-04-26T21:32:45.905520+0000","last_fullsized":"2024-04-26T21:32:45.905520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:24:50.476949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:47.885709+0000","last_change":"2024-04-26T21:32:43.833773+0000","last_active":"2024-04-26T21:32:47.885709+0000","last_peered":"2024-04-26T21:32:47.885709+0000","last_clean":"2024-04-26T21:32:47.885709+0000","last_became_active":"2024-04-26T21:32:43.833338+0000","last_became_peered":"2024-04-26T21:32:43.833338+0000","last_unstale":"2024-04-26T21:32:47.885709+0000","last_undegraded":"2024-04-26T21:32:47.885709+0000","last_fullsized":"2024-04-26T21:32:47.885709+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:15.382494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":336,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:46.094775+0000","last_change":"2024-04-26T21:31:27.930370+0000","last_active":"2024-04-26T21:32:46.094775+0000","last_peered":"2024-04-26T21:32:46.094775+0000","last_clean":"2024-04-26T21:32:46.094775+0000","last_became_active":"2024-04-26T21:31:27.929859+0000","last_became_peered":"2024-04-26T21:31:27.929859+0000","last_unstale":"2024-04-26T21:32:46.094775+0000","last_undegraded":"2024-04-26T21:32:46.094775+0000","last_fullsized":"2024-04-26T21:32:46.094775+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:58.528699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:58.528699+0000","last_clean_scrub_stamp":"2024-04-26T21:27:58.528699+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:00:55.701865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.850110+0000","last_change":"2024-04-26T21:32:44.858891+0000","last_active":"2024-04-26T21:32:45.850110+0000","last_peered":"2024-04-26T21:32:45.850110+0000","last_clean":"2024-04-26T21:32:45.850110+0000","last_became_active":"2024-04-26T21:32:44.858406+0000","last_became_peered":"2024-04-26T21:32:44.858406+0000","last_unstale":"2024-04-26T21:32:45.850110+0000","last_undegraded":"2024-04-26T21:32:45.850110+0000","last_fullsized":"2024-04-26T21:32:45.850110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:40:13.084352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:48.893527+0000","last_change":"2024-04-26T21:32:43.849442+0000","last_active":"2024-04-26T21:32:48.893527+0000","last_peered":"2024-04-26T21:32:48.893527+0000","last_clean":"2024-04-26T21:32:48.893527+0000","last_became_active":"2024-04-26T21:32:43.847951+0000","last_became_peered":"2024-04-26T21:32:43.847951+0000","last_unstale":"2024-04-26T21:32:48.893527+0000","last_undegraded":"2024-04-26T21:32:48.893527+0000","last_fullsized":"2024-04-26T21:32:48.893527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:55:00.025931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:47.884695+0000","last_change":"2024-04-26T21:32:43.846736+0000","last_active":"2024-04-26T21:32:47.884695+0000","last_peered":"2024-04-26T21:32:47.884695+0000","last_clean":"2024-04-26T21:32:47.884695+0000","last_became_active":"2024-04-26T21:32:43.846451+0000","last_became_peered":"2024-04-26T21:32:43.846451+0000","last_unstale":"2024-04-26T21:32:47.884695+0000","last_undegraded":"2024-04-26T21:32:47.884695+0000","last_fullsized":"2024-04-26T21:32:47.884695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:38:46.301597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.850352+0000","last_change":"2024-04-26T21:32:44.856817+0000","last_active":"2024-04-26T21:32:45.850352+0000","last_peered":"2024-04-26T21:32:45.850352+0000","last_clean":"2024-04-26T21:32:45.850352+0000","last_became_active":"2024-04-26T21:32:44.856489+0000","last_became_peered":"2024-04-26T21:32:44.856489+0000","last_unstale":"2024-04-26T21:32:45.850352+0000","last_undegraded":"2024-04-26T21:32:45.850352+0000","last_fullsized":"2024-04-26T21:32:45.850352+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:13:33.851351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.964813+0000","last_change":"2024-04-26T21:32:43.838974+0000","last_active":"2024-04-26T21:32:45.964813+0000","last_peered":"2024-04-26T21:32:45.964813+0000","last_clean":"2024-04-26T21:32:45.964813+0000","last_became_active":"2024-04-26T21:32:43.838661+0000","last_became_peered":"2024-04-26T21:32:43.838661+0000","last_unstale":"2024-04-26T21:32:45.964813+0000","last_undegraded":"2024-04-26T21:32:45.964813+0000","last_fullsized":"2024-04-26T21:32:45.964813+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:44:29.536559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905816+0000","last_change":"2024-04-26T21:32:44.872716+0000","last_active":"2024-04-26T21:32:45.905816+0000","last_peered":"2024-04-26T21:32:45.905816+0000","last_clean":"2024-04-26T21:32:45.905816+0000","last_became_active":"2024-04-26T21:32:44.872348+0000","last_became_peered":"2024-04-26T21:32:44.872348+0000","last_unstale":"2024-04-26T21:32:45.905816+0000","last_undegraded":"2024-04-26T21:32:45.905816+0000","last_fullsized":"2024-04-26T21:32:45.905816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:17:56.890488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:47.884204+0000","last_change":"2024-04-26T21:32:43.853138+0000","last_active":"2024-04-26T21:32:47.884204+0000","last_peered":"2024-04-26T21:32:47.884204+0000","last_clean":"2024-04-26T21:32:47.884204+0000","last_became_active":"2024-04-26T21:32:43.852803+0000","last_became_peered":"2024-04-26T21:32:43.852803+0000","last_unstale":"2024-04-26T21:32:47.884204+0000","last_undegraded":"2024-04-26T21:32:47.884204+0000","last_fullsized":"2024-04-26T21:32:47.884204+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:59:04.790442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904591+0000","last_change":"2024-04-26T21:32:44.867559+0000","last_active":"2024-04-26T21:32:45.904591+0000","last_peered":"2024-04-26T21:32:45.904591+0000","last_clean":"2024-04-26T21:32:45.904591+0000","last_became_active":"2024-04-26T21:32:44.867229+0000","last_became_peered":"2024-04-26T21:32:44.867229+0000","last_unstale":"2024-04-26T21:32:45.904591+0000","last_undegraded":"2024-04-26T21:32:45.904591+0000","last_fullsized":"2024-04-26T21:32:45.904591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:09:09.052753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:47.885134+0000","last_change":"2024-04-26T21:32:43.844761+0000","last_active":"2024-04-26T21:32:47.885134+0000","last_peered":"2024-04-26T21:32:47.885134+0000","last_clean":"2024-04-26T21:32:47.885134+0000","last_became_active":"2024-04-26T21:32:43.844257+0000","last_became_peered":"2024-04-26T21:32:43.844257+0000","last_unstale":"2024-04-26T21:32:47.885134+0000","last_undegraded":"2024-04-26T21:32:47.885134+0000","last_fullsized":"2024-04-26T21:32:47.885134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:06:55.560467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904670+0000","last_change":"2024-04-26T21:32:44.856662+0000","last_active":"2024-04-26T21:32:45.904670+0000","last_peered":"2024-04-26T21:32:45.904670+0000","last_clean":"2024-04-26T21:32:45.904670+0000","last_became_active":"2024-04-26T21:32:44.856297+0000","last_became_peered":"2024-04-26T21:32:44.856297+0000","last_unstale":"2024-04-26T21:32:45.904670+0000","last_undegraded":"2024-04-26T21:32:45.904670+0000","last_fullsized":"2024-04-26T21:32:45.904670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:48:55.314959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:47.884729+0000","last_change":"2024-04-26T21:32:43.848710+0000","last_active":"2024-04-26T21:32:47.884729+0000","last_peered":"2024-04-26T21:32:47.884729+0000","last_clean":"2024-04-26T21:32:47.884729+0000","last_became_active":"2024-04-26T21:32:43.848379+0000","last_became_peered":"2024-04-26T21:32:43.848379+0000","last_unstale":"2024-04-26T21:32:47.884729+0000","last_undegraded":"2024-04-26T21:32:47.884729+0000","last_fullsized":"2024-04-26T21:32:47.884729+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:26:44.233330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905408+0000","last_change":"2024-04-26T21:32:44.858327+0000","last_active":"2024-04-26T21:32:45.905408+0000","last_peered":"2024-04-26T21:32:45.905408+0000","last_clean":"2024-04-26T21:32:45.905408+0000","last_became_active":"2024-04-26T21:32:44.858023+0000","last_became_peered":"2024-04-26T21:32:44.858023+0000","last_unstale":"2024-04-26T21:32:45.905408+0000","last_undegraded":"2024-04-26T21:32:45.905408+0000","last_fullsized":"2024-04-26T21:32:45.905408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:30.588063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.851623+0000","last_change":"2024-04-26T21:32:43.839304+0000","last_active":"2024-04-26T21:32:45.851623+0000","last_peered":"2024-04-26T21:32:45.851623+0000","last_clean":"2024-04-26T21:32:45.851623+0000","last_became_active":"2024-04-26T21:32:43.838875+0000","last_became_peered":"2024-04-26T21:32:43.838875+0000","last_unstale":"2024-04-26T21:32:45.851623+0000","last_undegraded":"2024-04-26T21:32:45.851623+0000","last_fullsized":"2024-04-26T21:32:45.851623+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:07:48.761227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905769+0000","last_change":"2024-04-26T21:32:44.867088+0000","last_active":"2024-04-26T21:32:45.905769+0000","last_peered":"2024-04-26T21:32:45.905769+0000","last_clean":"2024-04-26T21:32:45.905769+0000","last_became_active":"2024-04-26T21:32:44.863623+0000","last_became_peered":"2024-04-26T21:32:44.863623+0000","last_unstale":"2024-04-26T21:32:45.905769+0000","last_undegraded":"2024-04-26T21:32:45.905769+0000","last_fullsized":"2024-04-26T21:32:45.905769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:42:21.107362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:47.884500+0000","last_change":"2024-04-26T21:32:43.844740+0000","last_active":"2024-04-26T21:32:47.884500+0000","last_peered":"2024-04-26T21:32:47.884500+0000","last_clean":"2024-04-26T21:32:47.884500+0000","last_became_active":"2024-04-26T21:32:43.844453+0000","last_became_peered":"2024-04-26T21:32:43.844453+0000","last_unstale":"2024-04-26T21:32:47.884500+0000","last_undegraded":"2024-04-26T21:32:47.884500+0000","last_fullsized":"2024-04-26T21:32:47.884500+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:34:02.264468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:46.095377+0000","last_change":"2024-04-26T21:32:44.863365+0000","last_active":"2024-04-26T21:32:46.095377+0000","last_peered":"2024-04-26T21:32:46.095377+0000","last_clean":"2024-04-26T21:32:46.095377+0000","last_became_active":"2024-04-26T21:32:44.863012+0000","last_became_peered":"2024-04-26T21:32:44.863012+0000","last_unstale":"2024-04-26T21:32:46.095377+0000","last_undegraded":"2024-04-26T21:32:46.095377+0000","last_fullsized":"2024-04-26T21:32:46.095377+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:09:50.023773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.852062+0000","last_change":"2024-04-26T21:32:43.853507+0000","last_active":"2024-04-26T21:32:45.852062+0000","last_peered":"2024-04-26T21:32:45.852062+0000","last_clean":"2024-04-26T21:32:45.852062+0000","last_became_active":"2024-04-26T21:32:43.852680+0000","last_became_peered":"2024-04-26T21:32:43.852680+0000","last_unstale":"2024-04-26T21:32:45.852062+0000","last_undegraded":"2024-04-26T21:32:45.852062+0000","last_fullsized":"2024-04-26T21:32:45.852062+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:46:51.836736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:46.095361+0000","last_change":"2024-04-26T21:32:44.865158+0000","last_active":"2024-04-26T21:32:46.095361+0000","last_peered":"2024-04-26T21:32:46.095361+0000","last_clean":"2024-04-26T21:32:46.095361+0000","last_became_active":"2024-04-26T21:32:44.864182+0000","last_became_peered":"2024-04-26T21:32:44.864182+0000","last_unstale":"2024-04-26T21:32:46.095361+0000","last_undegraded":"2024-04-26T21:32:46.095361+0000","last_fullsized":"2024-04-26T21:32:46.095361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:55.918623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905744+0000","last_change":"2024-04-26T21:32:43.842913+0000","last_active":"2024-04-26T21:32:45.905744+0000","last_peered":"2024-04-26T21:32:45.905744+0000","last_clean":"2024-04-26T21:32:45.905744+0000","last_became_active":"2024-04-26T21:32:43.842599+0000","last_became_peered":"2024-04-26T21:32:43.842599+0000","last_unstale":"2024-04-26T21:32:45.905744+0000","last_undegraded":"2024-04-26T21:32:45.905744+0000","last_fullsized":"2024-04-26T21:32:45.905744+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:46.583398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:46.095177+0000","last_change":"2024-04-26T21:32:44.864239+0000","last_active":"2024-04-26T21:32:46.095177+0000","last_peered":"2024-04-26T21:32:46.095177+0000","last_clean":"2024-04-26T21:32:46.095177+0000","last_became_active":"2024-04-26T21:32:44.863862+0000","last_became_peered":"2024-04-26T21:32:44.863862+0000","last_unstale":"2024-04-26T21:32:46.095177+0000","last_undegraded":"2024-04-26T21:32:46.095177+0000","last_fullsized":"2024-04-26T21:32:46.095177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:32:16.958044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.850363+0000","last_change":"2024-04-26T21:32:43.849768+0000","last_active":"2024-04-26T21:32:45.850363+0000","last_peered":"2024-04-26T21:32:45.850363+0000","last_clean":"2024-04-26T21:32:45.850363+0000","last_became_active":"2024-04-26T21:32:43.849424+0000","last_became_peered":"2024-04-26T21:32:43.849424+0000","last_unstale":"2024-04-26T21:32:45.850363+0000","last_undegraded":"2024-04-26T21:32:45.850363+0000","last_fullsized":"2024-04-26T21:32:45.850363+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:46:23.231819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:46.095209+0000","last_change":"2024-04-26T21:32:44.862597+0000","last_active":"2024-04-26T21:32:46.095209+0000","last_peered":"2024-04-26T21:32:46.095209+0000","last_clean":"2024-04-26T21:32:46.095209+0000","last_became_active":"2024-04-26T21:32:44.862259+0000","last_became_peered":"2024-04-26T21:32:44.862259+0000","last_unstale":"2024-04-26T21:32:46.095209+0000","last_undegraded":"2024-04-26T21:32:46.095209+0000","last_fullsized":"2024-04-26T21:32:46.095209+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:19.759329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905522+0000","last_change":"2024-04-26T21:32:44.855196+0000","last_active":"2024-04-26T21:32:45.905522+0000","last_peered":"2024-04-26T21:32:45.905522+0000","last_clean":"2024-04-26T21:32:45.905522+0000","last_became_active":"2024-04-26T21:32:44.854774+0000","last_became_peered":"2024-04-26T21:32:44.854774+0000","last_unstale":"2024-04-26T21:32:45.905522+0000","last_undegraded":"2024-04-26T21:32:45.905522+0000","last_fullsized":"2024-04-26T21:32:45.905522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:46:18.370983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.851774+0000","last_change":"2024-04-26T21:32:44.868115+0000","last_active":"2024-04-26T21:32:45.851774+0000","last_peered":"2024-04-26T21:32:45.851774+0000","last_clean":"2024-04-26T21:32:45.851774+0000","last_became_active":"2024-04-26T21:32:44.867767+0000","last_became_peered":"2024-04-26T21:32:44.867767+0000","last_unstale":"2024-04-26T21:32:45.851774+0000","last_undegraded":"2024-04-26T21:32:45.851774+0000","last_fullsized":"2024-04-26T21:32:45.851774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:30:22.130786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.851815+0000","last_change":"2024-04-26T21:32:44.864633+0000","last_active":"2024-04-26T21:32:45.851815+0000","last_peered":"2024-04-26T21:32:45.851815+0000","last_clean":"2024-04-26T21:32:45.851815+0000","last_became_active":"2024-04-26T21:32:44.864245+0000","last_became_peered":"2024-04-26T21:32:44.864245+0000","last_unstale":"2024-04-26T21:32:45.851815+0000","last_undegraded":"2024-04-26T21:32:45.851815+0000","last_fullsized":"2024-04-26T21:32:45.851815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:10:20.315396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904498+0000","last_change":"2024-04-26T21:32:44.858122+0000","last_active":"2024-04-26T21:32:45.904498+0000","last_peered":"2024-04-26T21:32:45.904498+0000","last_clean":"2024-04-26T21:32:45.904498+0000","last_became_active":"2024-04-26T21:32:44.857513+0000","last_became_peered":"2024-04-26T21:32:44.857513+0000","last_unstale":"2024-04-26T21:32:45.904498+0000","last_undegraded":"2024-04-26T21:32:45.904498+0000","last_fullsized":"2024-04-26T21:32:45.904498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:50:40.725215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.850435+0000","last_change":"2024-04-26T21:32:44.858684+0000","last_active":"2024-04-26T21:32:45.850435+0000","last_peered":"2024-04-26T21:32:45.850435+0000","last_clean":"2024-04-26T21:32:45.850435+0000","last_became_active":"2024-04-26T21:32:44.858298+0000","last_became_peered":"2024-04-26T21:32:44.858298+0000","last_unstale":"2024-04-26T21:32:45.850435+0000","last_undegraded":"2024-04-26T21:32:45.850435+0000","last_fullsized":"2024-04-26T21:32:45.850435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:52:00.512883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905022+0000","last_change":"2024-04-26T21:32:44.852073+0000","last_active":"2024-04-26T21:32:45.905022+0000","last_peered":"2024-04-26T21:32:45.905022+0000","last_clean":"2024-04-26T21:32:45.905022+0000","last_became_active":"2024-04-26T21:32:44.851717+0000","last_became_peered":"2024-04-26T21:32:44.851717+0000","last_unstale":"2024-04-26T21:32:45.905022+0000","last_undegraded":"2024-04-26T21:32:45.905022+0000","last_fullsized":"2024-04-26T21:32:45.905022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:27:37.599933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.852196+0000","last_change":"2024-04-26T21:32:44.859503+0000","last_active":"2024-04-26T21:32:45.852196+0000","last_peered":"2024-04-26T21:32:45.852196+0000","last_clean":"2024-04-26T21:32:45.852196+0000","last_became_active":"2024-04-26T21:32:44.859009+0000","last_became_peered":"2024-04-26T21:32:44.859009+0000","last_unstale":"2024-04-26T21:32:45.852196+0000","last_undegraded":"2024-04-26T21:32:45.852196+0000","last_fullsized":"2024-04-26T21:32:45.852196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:00:26.307567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905758+0000","last_change":"2024-04-26T21:32:44.875092+0000","last_active":"2024-04-26T21:32:45.905758+0000","last_peered":"2024-04-26T21:32:45.905758+0000","last_clean":"2024-04-26T21:32:45.905758+0000","last_became_active":"2024-04-26T21:32:44.874819+0000","last_became_peered":"2024-04-26T21:32:44.874819+0000","last_unstale":"2024-04-26T21:32:45.905758+0000","last_undegraded":"2024-04-26T21:32:45.905758+0000","last_fullsized":"2024-04-26T21:32:45.905758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:48:34.178656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905891+0000","last_change":"2024-04-26T21:32:44.857666+0000","last_active":"2024-04-26T21:32:45.905891+0000","last_peered":"2024-04-26T21:32:45.905891+0000","last_clean":"2024-04-26T21:32:45.905891+0000","last_became_active":"2024-04-26T21:32:44.857342+0000","last_became_peered":"2024-04-26T21:32:44.857342+0000","last_unstale":"2024-04-26T21:32:45.905891+0000","last_undegraded":"2024-04-26T21:32:45.905891+0000","last_fullsized":"2024-04-26T21:32:45.905891+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:05:15.139919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905141+0000","last_change":"2024-04-26T21:32:44.868571+0000","last_active":"2024-04-26T21:32:45.905141+0000","last_peered":"2024-04-26T21:32:45.905141+0000","last_clean":"2024-04-26T21:32:45.905141+0000","last_became_active":"2024-04-26T21:32:44.868191+0000","last_became_peered":"2024-04-26T21:32:44.868191+0000","last_unstale":"2024-04-26T21:32:45.905141+0000","last_undegraded":"2024-04-26T21:32:45.905141+0000","last_fullsized":"2024-04-26T21:32:45.905141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:35.329277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904913+0000","last_change":"2024-04-26T21:32:44.867823+0000","last_active":"2024-04-26T21:32:45.904913+0000","last_peered":"2024-04-26T21:32:45.904913+0000","last_clean":"2024-04-26T21:32:45.904913+0000","last_became_active":"2024-04-26T21:32:44.867496+0000","last_became_peered":"2024-04-26T21:32:44.867496+0000","last_unstale":"2024-04-26T21:32:45.904913+0000","last_undegraded":"2024-04-26T21:32:45.904913+0000","last_fullsized":"2024-04-26T21:32:45.904913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:41:47.484332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.850514+0000","last_change":"2024-04-26T21:32:44.864720+0000","last_active":"2024-04-26T21:32:45.850514+0000","last_peered":"2024-04-26T21:32:45.850514+0000","last_clean":"2024-04-26T21:32:45.850514+0000","last_became_active":"2024-04-26T21:32:44.864395+0000","last_became_peered":"2024-04-26T21:32:44.864395+0000","last_unstale":"2024-04-26T21:32:45.850514+0000","last_undegraded":"2024-04-26T21:32:45.850514+0000","last_fullsized":"2024-04-26T21:32:45.850514+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:25.416249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904758+0000","last_change":"2024-04-26T21:32:44.862415+0000","last_active":"2024-04-26T21:32:45.904758+0000","last_peered":"2024-04-26T21:32:45.904758+0000","last_clean":"2024-04-26T21:32:45.904758+0000","last_became_active":"2024-04-26T21:32:44.862141+0000","last_became_peered":"2024-04-26T21:32:44.862141+0000","last_unstale":"2024-04-26T21:32:45.904758+0000","last_undegraded":"2024-04-26T21:32:45.904758+0000","last_fullsized":"2024-04-26T21:32:45.904758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:52:55.919407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280902,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":105,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":21406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93,"ondisk_log_size":93,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383695,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":917504,"data_stored":439103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645331,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":1032079,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808856,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":917504,"data_stored":439103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070494,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":933888,"data_stored":439259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332131,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":1031463,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495656,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":438521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40400,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704752,"statfs":{"total":95995035648,"available":95953666048,"internally_reserved":0,"allocated":925696,"data_stored":440133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":439999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":929792,"data_stored":440167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36240,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708912,"statfs":{"total":95995035648,"available":95957925888,"internally_reserved":0,"allocated":925696,"data_stored":438715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920819,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36240,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708912,"statfs":{"total":95995035648,"available":95957925888,"internally_reserved":0,"allocated":925696,"data_stored":439165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757294,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":991232,"data_stored":1030365,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34089,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34089,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34089,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":8212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T21:32:52.718 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T21:32:52.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.855+0000 7fcfefd0a640 1 -- 172.21.15.46:0/2569656752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe8105da0 msgr2=0x7fcfe8108190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:52.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.855+0000 7fcfefd0a640 1 --2- 172.21.15.46:0/2569656752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe8105da0 0x7fcfe8108190 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8009900 tx=0x7fcfd802a410 comp rx=0 tx=0).stop 2024-04-26T21:32:52.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.855+0000 7fcfefd0a640 1 -- 172.21.15.46:0/2569656752 shutdown_connections 2024-04-26T21:32:52.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.855+0000 7fcfefd0a640 1 --2- 172.21.15.46:0/2569656752 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfe81086d0 0x7fcfe810aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.855+0000 7fcfefd0a640 1 --2- 172.21.15.46:0/2569656752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe8105da0 0x7fcfe8108190 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.855+0000 7fcfefd0a640 1 --2- 172.21.15.46:0/2569656752 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcfe8103470 0x7fcfe8105860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.855+0000 7fcfefd0a640 1 -- 172.21.15.46:0/2569656752 >> 172.21.15.46:0/2569656752 conn(0x7fcfe80fce20 msgr2=0x7fcfe80ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:52.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.855+0000 7fcfefd0a640 1 -- 172.21.15.46:0/2569656752 shutdown_connections 2024-04-26T21:32:52.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.855+0000 7fcfefd0a640 1 -- 172.21.15.46:0/2569656752 wait complete. 2024-04-26T21:32:52.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.856+0000 7fcfefd0a640 1 Processor -- start 2024-04-26T21:32:52.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.856+0000 7fcfefd0a640 1 -- start start 2024-04-26T21:32:52.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.856+0000 7fcfefd0a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfe8103470 0x7fcfe8196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:52.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.856+0000 7fcfefd0a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcfe8105da0 0x7fcfe8196fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:52.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.856+0000 7fcfefd0a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe81086d0 0x7fcfe81a67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:52.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.856+0000 7fcfefd0a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe810e4b0 con 0x7fcfe81086d0 2024-04-26T21:32:52.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.856+0000 7fcfefd0a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe810e330 con 0x7fcfe8103470 2024-04-26T21:32:52.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.856+0000 7fcfefd0a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe810e630 con 0x7fcfe8105da0 2024-04-26T21:32:52.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.857+0000 7fcfee2a7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe81086d0 0x7fcfe81a67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:52.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.857+0000 7fcfedaa6640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfe8103470 0x7fcfe8196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:52.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.857+0000 7fcfee2a7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe81086d0 0x7fcfe81a67e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49566/0 (socket says 172.21.15.46:49566) 2024-04-26T21:32:52.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.857+0000 7fcfee2a7640 1 -- 172.21.15.46:0/3776965531 learned_addr learned my addr 172.21.15.46:0/3776965531 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.857+0000 7fcfed2a5640 1 --2- 172.21.15.46:0/3776965531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcfe8105da0 0x7fcfe8196fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.857+0000 7fcfee2a7640 1 -- 172.21.15.46:0/3776965531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcfe8105da0 msgr2=0x7fcfe8196fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.857+0000 7fcfee2a7640 1 --2- 172.21.15.46:0/3776965531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcfe8105da0 0x7fcfe8196fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.857+0000 7fcfee2a7640 1 -- 172.21.15.46:0/3776965531 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfe8103470 msgr2=0x7fcfe8196a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.857+0000 7fcfee2a7640 1 --2- 172.21.15.46:0/3776965531 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfe8103470 0x7fcfe8196a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.857+0000 7fcfee2a7640 1 -- 172.21.15.46:0/3776965531 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfe81a6e50 con 0x7fcfe81086d0 2024-04-26T21:32:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.857+0000 7fcfee2a7640 1 --2- 172.21.15.46:0/3776965531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe81086d0 0x7fcfe81a67e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fcfe000eb10 tx=0x7fcfe000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.857+0000 7fcfdeffd640 1 -- 172.21.15.46:0/3776965531 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe0004b20 con 0x7fcfe81086d0 2024-04-26T21:32:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.857+0000 7fcfdeffd640 1 -- 172.21.15.46:0/3776965531 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfe0004cc0 con 0x7fcfe81086d0 2024-04-26T21:32:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.857+0000 7fcfefd0a640 1 -- 172.21.15.46:0/3776965531 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfe81a7140 con 0x7fcfe81086d0 2024-04-26T21:32:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.857+0000 7fcfefd0a640 1 -- 172.21.15.46:0/3776965531 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfe81a7610 con 0x7fcfe81086d0 2024-04-26T21:32:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.857+0000 7fcfdeffd640 1 -- 172.21.15.46:0/3776965531 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe0010670 con 0x7fcfe81086d0 2024-04-26T21:32:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.858+0000 7fcfefd0a640 1 -- 172.21.15.46:0/3776965531 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfe810a1c0 con 0x7fcfe81086d0 2024-04-26T21:32:52.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.860+0000 7fcfdeffd640 1 -- 172.21.15.46:0/3776965531 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fcfe0005020 con 0x7fcfe81086d0 2024-04-26T21:32:52.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.860+0000 7fcfdeffd640 1 --2- 172.21.15.46:0/3776965531 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcfd40a3bb0 0x7fcfd40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:52.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.860+0000 7fcfedaa6640 1 --2- 172.21.15.46:0/3776965531 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcfd40a3bb0 0x7fcfd40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:52.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.860+0000 7fcfdeffd640 1 -- 172.21.15.46:0/3776965531 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fcfe00d0860 con 0x7fcfe81086d0 2024-04-26T21:32:52.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.861+0000 7fcfedaa6640 1 --2- 172.21.15.46:0/3776965531 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcfd40a3bb0 0x7fcfd40a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4009bd0 tx=0x7fcfe4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:52.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.866+0000 7fcfdeffd640 1 -- 172.21.15.46:0/3776965531 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcfe009d390 con 0x7fcfe81086d0 2024-04-26T21:32:52.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:52.911+0000 7fcfdeffd640 1 -- 172.21.15.46:0/3776965531 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcfe0099300 con 0x7fcfe81086d0 2024-04-26T21:32:53.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:52 smithi046 ceph-mon[29201]: pgmap v296: 97 pgs: 97 active+clean; 581 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 4.6 KiB/s wr, 13 op/s 2024-04-26T21:32:53.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:52 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3166015083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:53.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:52 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1925659073' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:32:53.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:52 smithi046 ceph-mon[29201]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:32:53.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:53.061+0000 7fcfefd0a640 1 -- 172.21.15.46:0/3776965531 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcfe8062d70 con 0x7fcfd40a3bb0 2024-04-26T21:32:53.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:53.068+0000 7fcfdeffd640 1 -- 172.21.15.46:0/3776965531 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258901 (secure 0 0 0) 0x7fcfe8062d70 con 0x7fcfd40a3bb0 2024-04-26T21:32:53.073 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:32:53.076 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-04-26T21:32:53.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:53.071+0000 7fcfefd0a640 1 -- 172.21.15.46:0/3776965531 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcfd40a3bb0 msgr2=0x7fcfd40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:53.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:53.071+0000 7fcfefd0a640 1 --2- 172.21.15.46:0/3776965531 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcfd40a3bb0 0x7fcfd40a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4009bd0 tx=0x7fcfe4005f90 comp rx=0 tx=0).stop 2024-04-26T21:32:53.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:53.071+0000 7fcfefd0a640 1 -- 172.21.15.46:0/3776965531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe81086d0 msgr2=0x7fcfe81a67e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:53.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:53.071+0000 7fcfefd0a640 1 --2- 172.21.15.46:0/3776965531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe81086d0 0x7fcfe81a67e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fcfe000eb10 tx=0x7fcfe000c2d0 comp rx=0 tx=0).stop 2024-04-26T21:32:53.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:53.071+0000 7fcfefd0a640 1 -- 172.21.15.46:0/3776965531 shutdown_connections 2024-04-26T21:32:53.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:53.071+0000 7fcfefd0a640 1 --2- 172.21.15.46:0/3776965531 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcfd40a3bb0 0x7fcfd40a6070 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:53.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:53.072+0000 7fcfefd0a640 1 --2- 172.21.15.46:0/3776965531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcfe81086d0 0x7fcfe81a67e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:53.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:53.072+0000 7fcfefd0a640 1 --2- 172.21.15.46:0/3776965531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcfe8105da0 0x7fcfe8196fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:53.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:53.072+0000 7fcfefd0a640 1 --2- 172.21.15.46:0/3776965531 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfe8103470 0x7fcfe8196a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:53.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:53.072+0000 7fcfefd0a640 1 -- 172.21.15.46:0/3776965531 >> 172.21.15.46:0/3776965531 conn(0x7fcfe80fce20 msgr2=0x7fcfe8102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:53.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:53.072+0000 7fcfefd0a640 1 -- 172.21.15.46:0/3776965531 shutdown_connections 2024-04-26T21:32:53.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:53.072+0000 7fcfefd0a640 1 -- 172.21.15.46:0/3776965531 wait complete. 2024-04-26T21:32:53.084 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-04-26T21:32:51.931078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":131,"num_read_kb":114,"num_write":286,"num_write_kb":2975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":381020,"kb_used_data":11044,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124560804,"statfs":{"total":1151940427776,"available":1151550263296,"internally_reserved":0,"allocated":11309056,"data_stored":7048072,"data_compressed":107181,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19140,"internal_metadata":378713404},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4640,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":105,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001775"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905654+0000","last_change":"2024-04-26T21:32:44.863556+0000","last_active":"2024-04-26T21:32:45.905654+0000","last_peered":"2024-04-26T21:32:45.905654+0000","last_clean":"2024-04-26T21:32:45.905654+0000","last_became_active":"2024-04-26T21:32:44.863217+0000","last_became_peered":"2024-04-26T21:32:44.863217+0000","last_unstale":"2024-04-26T21:32:45.905654+0000","last_undegraded":"2024-04-26T21:32:45.905654+0000","last_fullsized":"2024-04-26T21:32:45.905654+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:36:19.416660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904461+0000","last_change":"2024-04-26T21:32:44.862340+0000","last_active":"2024-04-26T21:32:45.904461+0000","last_peered":"2024-04-26T21:32:45.904461+0000","last_clean":"2024-04-26T21:32:45.904461+0000","last_became_active":"2024-04-26T21:32:44.861753+0000","last_became_peered":"2024-04-26T21:32:44.861753+0000","last_unstale":"2024-04-26T21:32:45.904461+0000","last_undegraded":"2024-04-26T21:32:45.904461+0000","last_fullsized":"2024-04-26T21:32:45.904461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:56:46.395806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904731+0000","last_change":"2024-04-26T21:32:44.865678+0000","last_active":"2024-04-26T21:32:45.904731+0000","last_peered":"2024-04-26T21:32:45.904731+0000","last_clean":"2024-04-26T21:32:45.904731+0000","last_became_active":"2024-04-26T21:32:44.865001+0000","last_became_peered":"2024-04-26T21:32:44.865001+0000","last_unstale":"2024-04-26T21:32:45.904731+0000","last_undegraded":"2024-04-26T21:32:45.904731+0000","last_fullsized":"2024-04-26T21:32:45.904731+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:29:42.803401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905389+0000","last_change":"2024-04-26T21:32:44.869469+0000","last_active":"2024-04-26T21:32:45.905389+0000","last_peered":"2024-04-26T21:32:45.905389+0000","last_clean":"2024-04-26T21:32:45.905389+0000","last_became_active":"2024-04-26T21:32:44.868443+0000","last_became_peered":"2024-04-26T21:32:44.868443+0000","last_unstale":"2024-04-26T21:32:45.905389+0000","last_undegraded":"2024-04-26T21:32:45.905389+0000","last_fullsized":"2024-04-26T21:32:45.905389+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:41:43.252283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905138+0000","last_change":"2024-04-26T21:32:44.855500+0000","last_active":"2024-04-26T21:32:45.905138+0000","last_peered":"2024-04-26T21:32:45.905138+0000","last_clean":"2024-04-26T21:32:45.905138+0000","last_became_active":"2024-04-26T21:32:44.854902+0000","last_became_peered":"2024-04-26T21:32:44.854902+0000","last_unstale":"2024-04-26T21:32:45.905138+0000","last_undegraded":"2024-04-26T21:32:45.905138+0000","last_fullsized":"2024-04-26T21:32:45.905138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:14:02.135590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.851865+0000","last_change":"2024-04-26T21:32:44.864742+0000","last_active":"2024-04-26T21:32:45.851865+0000","last_peered":"2024-04-26T21:32:45.851865+0000","last_clean":"2024-04-26T21:32:45.851865+0000","last_became_active":"2024-04-26T21:32:44.864369+0000","last_became_peered":"2024-04-26T21:32:44.864369+0000","last_unstale":"2024-04-26T21:32:45.851865+0000","last_undegraded":"2024-04-26T21:32:45.851865+0000","last_fullsized":"2024-04-26T21:32:45.851865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:24:43.757233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:46.094853+0000","last_change":"2024-04-26T21:32:44.863158+0000","last_active":"2024-04-26T21:32:46.094853+0000","last_peered":"2024-04-26T21:32:46.094853+0000","last_clean":"2024-04-26T21:32:46.094853+0000","last_became_active":"2024-04-26T21:32:44.862773+0000","last_became_peered":"2024-04-26T21:32:44.862773+0000","last_unstale":"2024-04-26T21:32:46.094853+0000","last_undegraded":"2024-04-26T21:32:46.094853+0000","last_fullsized":"2024-04-26T21:32:46.094853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:15:44.907270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904664+0000","last_change":"2024-04-26T21:32:44.864585+0000","last_active":"2024-04-26T21:32:45.904664+0000","last_peered":"2024-04-26T21:32:45.904664+0000","last_clean":"2024-04-26T21:32:45.904664+0000","last_became_active":"2024-04-26T21:32:44.863572+0000","last_became_peered":"2024-04-26T21:32:44.863572+0000","last_unstale":"2024-04-26T21:32:45.904664+0000","last_undegraded":"2024-04-26T21:32:45.904664+0000","last_fullsized":"2024-04-26T21:32:45.904664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:32:42.481698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905268+0000","last_change":"2024-04-26T21:32:44.855339+0000","last_active":"2024-04-26T21:32:45.905268+0000","last_peered":"2024-04-26T21:32:45.905268+0000","last_clean":"2024-04-26T21:32:45.905268+0000","last_became_active":"2024-04-26T21:32:44.853923+0000","last_became_peered":"2024-04-26T21:32:44.853923+0000","last_unstale":"2024-04-26T21:32:45.905268+0000","last_undegraded":"2024-04-26T21:32:45.905268+0000","last_fullsized":"2024-04-26T21:32:45.905268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:08:03.274120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.851852+0000","last_change":"2024-04-26T21:32:44.865564+0000","last_active":"2024-04-26T21:32:45.851852+0000","last_peered":"2024-04-26T21:32:45.851852+0000","last_clean":"2024-04-26T21:32:45.851852+0000","last_became_active":"2024-04-26T21:32:44.865209+0000","last_became_peered":"2024-04-26T21:32:44.865209+0000","last_unstale":"2024-04-26T21:32:45.851852+0000","last_undegraded":"2024-04-26T21:32:45.851852+0000","last_fullsized":"2024-04-26T21:32:45.851852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:51:36.388624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:46.095218+0000","last_change":"2024-04-26T21:32:44.865437+0000","last_active":"2024-04-26T21:32:46.095218+0000","last_peered":"2024-04-26T21:32:46.095218+0000","last_clean":"2024-04-26T21:32:46.095218+0000","last_became_active":"2024-04-26T21:32:44.864662+0000","last_became_peered":"2024-04-26T21:32:44.864662+0000","last_unstale":"2024-04-26T21:32:46.095218+0000","last_undegraded":"2024-04-26T21:32:46.095218+0000","last_fullsized":"2024-04-26T21:32:46.095218+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:23:37.867338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905241+0000","last_change":"2024-04-26T21:32:44.869738+0000","last_active":"2024-04-26T21:32:45.905241+0000","last_peered":"2024-04-26T21:32:45.905241+0000","last_clean":"2024-04-26T21:32:45.905241+0000","last_became_active":"2024-04-26T21:32:44.869133+0000","last_became_peered":"2024-04-26T21:32:44.869133+0000","last_unstale":"2024-04-26T21:32:45.905241+0000","last_undegraded":"2024-04-26T21:32:45.905241+0000","last_fullsized":"2024-04-26T21:32:45.905241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:08:29.233128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.850576+0000","last_change":"2024-04-26T21:32:44.856970+0000","last_active":"2024-04-26T21:32:45.850576+0000","last_peered":"2024-04-26T21:32:45.850576+0000","last_clean":"2024-04-26T21:32:45.850576+0000","last_became_active":"2024-04-26T21:32:44.856579+0000","last_became_peered":"2024-04-26T21:32:44.856579+0000","last_unstale":"2024-04-26T21:32:45.850576+0000","last_undegraded":"2024-04-26T21:32:45.850576+0000","last_fullsized":"2024-04-26T21:32:45.850576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:24:17.281094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904282+0000","last_change":"2024-04-26T21:32:44.859688+0000","last_active":"2024-04-26T21:32:45.904282+0000","last_peered":"2024-04-26T21:32:45.904282+0000","last_clean":"2024-04-26T21:32:45.904282+0000","last_became_active":"2024-04-26T21:32:44.859336+0000","last_became_peered":"2024-04-26T21:32:44.859336+0000","last_unstale":"2024-04-26T21:32:45.904282+0000","last_undegraded":"2024-04-26T21:32:45.904282+0000","last_fullsized":"2024-04-26T21:32:45.904282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:55:46.956334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904017+0000","last_change":"2024-04-26T21:32:44.860647+0000","last_active":"2024-04-26T21:32:45.904017+0000","last_peered":"2024-04-26T21:32:45.904017+0000","last_clean":"2024-04-26T21:32:45.904017+0000","last_became_active":"2024-04-26T21:32:44.860237+0000","last_became_peered":"2024-04-26T21:32:44.860237+0000","last_unstale":"2024-04-26T21:32:45.904017+0000","last_undegraded":"2024-04-26T21:32:45.904017+0000","last_fullsized":"2024-04-26T21:32:45.904017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:49:10.042990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905459+0000","last_change":"2024-04-26T21:32:44.865289+0000","last_active":"2024-04-26T21:32:45.905459+0000","last_peered":"2024-04-26T21:32:45.905459+0000","last_clean":"2024-04-26T21:32:45.905459+0000","last_became_active":"2024-04-26T21:32:44.864951+0000","last_became_peered":"2024-04-26T21:32:44.864951+0000","last_unstale":"2024-04-26T21:32:45.905459+0000","last_undegraded":"2024-04-26T21:32:45.905459+0000","last_fullsized":"2024-04-26T21:32:45.905459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:31:51.998634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905976+0000","last_change":"2024-04-26T21:32:44.857616+0000","last_active":"2024-04-26T21:32:45.905976+0000","last_peered":"2024-04-26T21:32:45.905976+0000","last_clean":"2024-04-26T21:32:45.905976+0000","last_became_active":"2024-04-26T21:32:44.857259+0000","last_became_peered":"2024-04-26T21:32:44.857259+0000","last_unstale":"2024-04-26T21:32:45.905976+0000","last_undegraded":"2024-04-26T21:32:45.905976+0000","last_fullsized":"2024-04-26T21:32:45.905976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:59:37.412657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905236+0000","last_change":"2024-04-26T21:32:44.857177+0000","last_active":"2024-04-26T21:32:45.905236+0000","last_peered":"2024-04-26T21:32:45.905236+0000","last_clean":"2024-04-26T21:32:45.905236+0000","last_became_active":"2024-04-26T21:32:44.856810+0000","last_became_peered":"2024-04-26T21:32:44.856810+0000","last_unstale":"2024-04-26T21:32:45.905236+0000","last_undegraded":"2024-04-26T21:32:45.905236+0000","last_fullsized":"2024-04-26T21:32:45.905236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:31:40.712896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.851995+0000","last_change":"2024-04-26T21:32:44.870498+0000","last_active":"2024-04-26T21:32:45.851995+0000","last_peered":"2024-04-26T21:32:45.851995+0000","last_clean":"2024-04-26T21:32:45.851995+0000","last_became_active":"2024-04-26T21:32:44.870170+0000","last_became_peered":"2024-04-26T21:32:44.870170+0000","last_unstale":"2024-04-26T21:32:45.851995+0000","last_undegraded":"2024-04-26T21:32:45.851995+0000","last_fullsized":"2024-04-26T21:32:45.851995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:27.428027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905364+0000","last_change":"2024-04-26T21:32:43.845192+0000","last_active":"2024-04-26T21:32:45.905364+0000","last_peered":"2024-04-26T21:32:45.905364+0000","last_clean":"2024-04-26T21:32:45.905364+0000","last_became_active":"2024-04-26T21:32:43.844862+0000","last_became_peered":"2024-04-26T21:32:43.844862+0000","last_unstale":"2024-04-26T21:32:45.905364+0000","last_undegraded":"2024-04-26T21:32:45.905364+0000","last_fullsized":"2024-04-26T21:32:45.905364+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:07:00.060047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905637+0000","last_change":"2024-04-26T21:32:44.867358+0000","last_active":"2024-04-26T21:32:45.905637+0000","last_peered":"2024-04-26T21:32:45.905637+0000","last_clean":"2024-04-26T21:32:45.905637+0000","last_became_active":"2024-04-26T21:32:44.864833+0000","last_became_peered":"2024-04-26T21:32:44.864833+0000","last_unstale":"2024-04-26T21:32:45.905637+0000","last_undegraded":"2024-04-26T21:32:45.905637+0000","last_fullsized":"2024-04-26T21:32:45.905637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:45:50.285471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905648+0000","last_change":"2024-04-26T21:32:43.849724+0000","last_active":"2024-04-26T21:32:45.905648+0000","last_peered":"2024-04-26T21:32:45.905648+0000","last_clean":"2024-04-26T21:32:45.905648+0000","last_became_active":"2024-04-26T21:32:43.849040+0000","last_became_peered":"2024-04-26T21:32:43.849040+0000","last_unstale":"2024-04-26T21:32:45.905648+0000","last_undegraded":"2024-04-26T21:32:45.905648+0000","last_fullsized":"2024-04-26T21:32:45.905648+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:48.994093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904631+0000","last_change":"2024-04-26T21:32:44.863095+0000","last_active":"2024-04-26T21:32:45.904631+0000","last_peered":"2024-04-26T21:32:45.904631+0000","last_clean":"2024-04-26T21:32:45.904631+0000","last_became_active":"2024-04-26T21:32:44.862739+0000","last_became_peered":"2024-04-26T21:32:44.862739+0000","last_unstale":"2024-04-26T21:32:45.904631+0000","last_undegraded":"2024-04-26T21:32:45.904631+0000","last_fullsized":"2024-04-26T21:32:45.904631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:21:19.927248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:48.894281+0000","last_change":"2024-04-26T21:32:43.854315+0000","last_active":"2024-04-26T21:32:48.894281+0000","last_peered":"2024-04-26T21:32:48.894281+0000","last_clean":"2024-04-26T21:32:48.894281+0000","last_became_active":"2024-04-26T21:32:43.849472+0000","last_became_peered":"2024-04-26T21:32:43.849472+0000","last_unstale":"2024-04-26T21:32:48.894281+0000","last_undegraded":"2024-04-26T21:32:48.894281+0000","last_fullsized":"2024-04-26T21:32:48.894281+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:42:24.213303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904901+0000","last_change":"2024-04-26T21:32:44.860100+0000","last_active":"2024-04-26T21:32:45.904901+0000","last_peered":"2024-04-26T21:32:45.904901+0000","last_clean":"2024-04-26T21:32:45.904901+0000","last_became_active":"2024-04-26T21:32:44.858890+0000","last_became_peered":"2024-04-26T21:32:44.858890+0000","last_unstale":"2024-04-26T21:32:45.904901+0000","last_undegraded":"2024-04-26T21:32:45.904901+0000","last_fullsized":"2024-04-26T21:32:45.904901+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:22.749764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905078+0000","last_change":"2024-04-26T21:32:43.840719+0000","last_active":"2024-04-26T21:32:45.905078+0000","last_peered":"2024-04-26T21:32:45.905078+0000","last_clean":"2024-04-26T21:32:45.905078+0000","last_became_active":"2024-04-26T21:32:43.840205+0000","last_became_peered":"2024-04-26T21:32:43.840205+0000","last_unstale":"2024-04-26T21:32:45.905078+0000","last_undegraded":"2024-04-26T21:32:45.905078+0000","last_fullsized":"2024-04-26T21:32:45.905078+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:29:41.025573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905966+0000","last_change":"2024-04-26T21:32:44.858287+0000","last_active":"2024-04-26T21:32:45.905966+0000","last_peered":"2024-04-26T21:32:45.905966+0000","last_clean":"2024-04-26T21:32:45.905966+0000","last_became_active":"2024-04-26T21:32:44.857890+0000","last_became_peered":"2024-04-26T21:32:44.857890+0000","last_unstale":"2024-04-26T21:32:45.905966+0000","last_undegraded":"2024-04-26T21:32:45.905966+0000","last_fullsized":"2024-04-26T21:32:45.905966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:16:23.743633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:47.886087+0000","last_change":"2024-04-26T21:32:43.850841+0000","last_active":"2024-04-26T21:32:47.886087+0000","last_peered":"2024-04-26T21:32:47.886087+0000","last_clean":"2024-04-26T21:32:47.886087+0000","last_became_active":"2024-04-26T21:32:43.850512+0000","last_became_peered":"2024-04-26T21:32:43.850512+0000","last_unstale":"2024-04-26T21:32:47.886087+0000","last_undegraded":"2024-04-26T21:32:47.886087+0000","last_fullsized":"2024-04-26T21:32:47.886087+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:13:51.814145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905362+0000","last_change":"2024-04-26T21:32:44.853438+0000","last_active":"2024-04-26T21:32:45.905362+0000","last_peered":"2024-04-26T21:32:45.905362+0000","last_clean":"2024-04-26T21:32:45.905362+0000","last_became_active":"2024-04-26T21:32:44.853126+0000","last_became_peered":"2024-04-26T21:32:44.853126+0000","last_unstale":"2024-04-26T21:32:45.905362+0000","last_undegraded":"2024-04-26T21:32:45.905362+0000","last_fullsized":"2024-04-26T21:32:45.905362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:43.054626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:48.895433+0000","last_change":"2024-04-26T21:32:43.839230+0000","last_active":"2024-04-26T21:32:48.895433+0000","last_peered":"2024-04-26T21:32:48.895433+0000","last_clean":"2024-04-26T21:32:48.895433+0000","last_became_active":"2024-04-26T21:32:43.838894+0000","last_became_peered":"2024-04-26T21:32:43.838894+0000","last_unstale":"2024-04-26T21:32:48.895433+0000","last_undegraded":"2024-04-26T21:32:48.895433+0000","last_fullsized":"2024-04-26T21:32:48.895433+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:54:29.718314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905045+0000","last_change":"2024-04-26T21:32:44.859870+0000","last_active":"2024-04-26T21:32:45.905045+0000","last_peered":"2024-04-26T21:32:45.905045+0000","last_clean":"2024-04-26T21:32:45.905045+0000","last_became_active":"2024-04-26T21:32:44.859474+0000","last_became_peered":"2024-04-26T21:32:44.859474+0000","last_unstale":"2024-04-26T21:32:45.905045+0000","last_undegraded":"2024-04-26T21:32:45.905045+0000","last_fullsized":"2024-04-26T21:32:45.905045+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:13:14.854523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:48.893992+0000","last_change":"2024-04-26T21:32:43.841090+0000","last_active":"2024-04-26T21:32:48.893992+0000","last_peered":"2024-04-26T21:32:48.893992+0000","last_clean":"2024-04-26T21:32:48.893992+0000","last_became_active":"2024-04-26T21:32:43.840723+0000","last_became_peered":"2024-04-26T21:32:43.840723+0000","last_unstale":"2024-04-26T21:32:48.893992+0000","last_undegraded":"2024-04-26T21:32:48.893992+0000","last_fullsized":"2024-04-26T21:32:48.893992+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:36:45.799738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904881+0000","last_change":"2024-04-26T21:32:44.869439+0000","last_active":"2024-04-26T21:32:45.904881+0000","last_peered":"2024-04-26T21:32:45.904881+0000","last_clean":"2024-04-26T21:32:45.904881+0000","last_became_active":"2024-04-26T21:32:44.868696+0000","last_became_peered":"2024-04-26T21:32:44.868696+0000","last_unstale":"2024-04-26T21:32:45.904881+0000","last_undegraded":"2024-04-26T21:32:45.904881+0000","last_fullsized":"2024-04-26T21:32:45.904881+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:46.400937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905617+0000","last_change":"2024-04-26T21:32:43.852431+0000","last_active":"2024-04-26T21:32:45.905617+0000","last_peered":"2024-04-26T21:32:45.905617+0000","last_clean":"2024-04-26T21:32:45.905617+0000","last_became_active":"2024-04-26T21:32:43.852143+0000","last_became_peered":"2024-04-26T21:32:43.852143+0000","last_unstale":"2024-04-26T21:32:45.905617+0000","last_undegraded":"2024-04-26T21:32:45.905617+0000","last_fullsized":"2024-04-26T21:32:45.905617+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:35.971582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904878+0000","last_change":"2024-04-26T21:32:44.861118+0000","last_active":"2024-04-26T21:32:45.904878+0000","last_peered":"2024-04-26T21:32:45.904878+0000","last_clean":"2024-04-26T21:32:45.904878+0000","last_became_active":"2024-04-26T21:32:44.860765+0000","last_became_peered":"2024-04-26T21:32:44.860765+0000","last_unstale":"2024-04-26T21:32:45.904878+0000","last_undegraded":"2024-04-26T21:32:45.904878+0000","last_fullsized":"2024-04-26T21:32:45.904878+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:01:09.370401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:47.885238+0000","last_change":"2024-04-26T21:32:43.840267+0000","last_active":"2024-04-26T21:32:47.885238+0000","last_peered":"2024-04-26T21:32:47.885238+0000","last_clean":"2024-04-26T21:32:47.885238+0000","last_became_active":"2024-04-26T21:32:43.839944+0000","last_became_peered":"2024-04-26T21:32:43.839944+0000","last_unstale":"2024-04-26T21:32:47.885238+0000","last_undegraded":"2024-04-26T21:32:47.885238+0000","last_fullsized":"2024-04-26T21:32:47.885238+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:46:13.671063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905332+0000","last_change":"2024-04-26T21:32:44.861772+0000","last_active":"2024-04-26T21:32:45.905332+0000","last_peered":"2024-04-26T21:32:45.905332+0000","last_clean":"2024-04-26T21:32:45.905332+0000","last_became_active":"2024-04-26T21:32:44.860376+0000","last_became_peered":"2024-04-26T21:32:44.860376+0000","last_unstale":"2024-04-26T21:32:45.905332+0000","last_undegraded":"2024-04-26T21:32:45.905332+0000","last_fullsized":"2024-04-26T21:32:45.905332+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:07:29.499706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:47.885591+0000","last_change":"2024-04-26T21:32:43.849923+0000","last_active":"2024-04-26T21:32:47.885591+0000","last_peered":"2024-04-26T21:32:47.885591+0000","last_clean":"2024-04-26T21:32:47.885591+0000","last_became_active":"2024-04-26T21:32:43.848193+0000","last_became_peered":"2024-04-26T21:32:43.848193+0000","last_unstale":"2024-04-26T21:32:47.885591+0000","last_undegraded":"2024-04-26T21:32:47.885591+0000","last_fullsized":"2024-04-26T21:32:47.885591+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:06:00.247084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904552+0000","last_change":"2024-04-26T21:32:44.866684+0000","last_active":"2024-04-26T21:32:45.904552+0000","last_peered":"2024-04-26T21:32:45.904552+0000","last_clean":"2024-04-26T21:32:45.904552+0000","last_became_active":"2024-04-26T21:32:44.866350+0000","last_became_peered":"2024-04-26T21:32:44.866350+0000","last_unstale":"2024-04-26T21:32:45.904552+0000","last_undegraded":"2024-04-26T21:32:45.904552+0000","last_fullsized":"2024-04-26T21:32:45.904552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:21.306500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.906114+0000","last_change":"2024-04-26T21:32:43.841617+0000","last_active":"2024-04-26T21:32:45.906114+0000","last_peered":"2024-04-26T21:32:45.906114+0000","last_clean":"2024-04-26T21:32:45.906114+0000","last_became_active":"2024-04-26T21:32:43.841302+0000","last_became_peered":"2024-04-26T21:32:43.841302+0000","last_unstale":"2024-04-26T21:32:45.906114+0000","last_undegraded":"2024-04-26T21:32:45.906114+0000","last_fullsized":"2024-04-26T21:32:45.906114+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:16:25.409255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905147+0000","last_change":"2024-04-26T21:32:44.860793+0000","last_active":"2024-04-26T21:32:45.905147+0000","last_peered":"2024-04-26T21:32:45.905147+0000","last_clean":"2024-04-26T21:32:45.905147+0000","last_became_active":"2024-04-26T21:32:44.859418+0000","last_became_peered":"2024-04-26T21:32:44.859418+0000","last_unstale":"2024-04-26T21:32:45.905147+0000","last_undegraded":"2024-04-26T21:32:45.905147+0000","last_fullsized":"2024-04-26T21:32:45.905147+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:17:46.189899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'8","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:47.884595+0000","last_change":"2024-04-26T21:32:43.843122+0000","last_active":"2024-04-26T21:32:47.884595+0000","last_peered":"2024-04-26T21:32:47.884595+0000","last_clean":"2024-04-26T21:32:47.884595+0000","last_became_active":"2024-04-26T21:32:43.842751+0000","last_became_peered":"2024-04-26T21:32:43.842751+0000","last_unstale":"2024-04-26T21:32:47.884595+0000","last_undegraded":"2024-04-26T21:32:47.884595+0000","last_fullsized":"2024-04-26T21:32:47.884595+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:16:58.533030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905144+0000","last_change":"2024-04-26T21:32:44.872829+0000","last_active":"2024-04-26T21:32:45.905144+0000","last_peered":"2024-04-26T21:32:45.905144+0000","last_clean":"2024-04-26T21:32:45.905144+0000","last_became_active":"2024-04-26T21:32:44.872466+0000","last_became_peered":"2024-04-26T21:32:44.872466+0000","last_unstale":"2024-04-26T21:32:45.905144+0000","last_undegraded":"2024-04-26T21:32:45.905144+0000","last_fullsized":"2024-04-26T21:32:45.905144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:07:27.698789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.851758+0000","last_change":"2024-04-26T21:32:43.840277+0000","last_active":"2024-04-26T21:32:45.851758+0000","last_peered":"2024-04-26T21:32:45.851758+0000","last_clean":"2024-04-26T21:32:45.851758+0000","last_became_active":"2024-04-26T21:32:43.839962+0000","last_became_peered":"2024-04-26T21:32:43.839962+0000","last_unstale":"2024-04-26T21:32:45.851758+0000","last_undegraded":"2024-04-26T21:32:45.851758+0000","last_fullsized":"2024-04-26T21:32:45.851758+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:25:54.475216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.850259+0000","last_change":"2024-04-26T21:32:44.866465+0000","last_active":"2024-04-26T21:32:45.850259+0000","last_peered":"2024-04-26T21:32:45.850259+0000","last_clean":"2024-04-26T21:32:45.850259+0000","last_became_active":"2024-04-26T21:32:44.866067+0000","last_became_peered":"2024-04-26T21:32:44.866067+0000","last_unstale":"2024-04-26T21:32:45.850259+0000","last_undegraded":"2024-04-26T21:32:45.850259+0000","last_fullsized":"2024-04-26T21:32:45.850259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:47:03.884436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:47.886078+0000","last_change":"2024-04-26T21:32:43.851535+0000","last_active":"2024-04-26T21:32:47.886078+0000","last_peered":"2024-04-26T21:32:47.886078+0000","last_clean":"2024-04-26T21:32:47.886078+0000","last_became_active":"2024-04-26T21:32:43.851237+0000","last_became_peered":"2024-04-26T21:32:43.851237+0000","last_unstale":"2024-04-26T21:32:47.886078+0000","last_undegraded":"2024-04-26T21:32:47.886078+0000","last_fullsized":"2024-04-26T21:32:47.886078+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:08:40.624027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:46.095216+0000","last_change":"2024-04-26T21:32:44.874686+0000","last_active":"2024-04-26T21:32:46.095216+0000","last_peered":"2024-04-26T21:32:46.095216+0000","last_clean":"2024-04-26T21:32:46.095216+0000","last_became_active":"2024-04-26T21:32:44.874377+0000","last_became_peered":"2024-04-26T21:32:44.874377+0000","last_unstale":"2024-04-26T21:32:46.095216+0000","last_undegraded":"2024-04-26T21:32:46.095216+0000","last_fullsized":"2024-04-26T21:32:46.095216+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:45:33.627663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:48.894431+0000","last_change":"2024-04-26T21:32:43.846522+0000","last_active":"2024-04-26T21:32:48.894431+0000","last_peered":"2024-04-26T21:32:48.894431+0000","last_clean":"2024-04-26T21:32:48.894431+0000","last_became_active":"2024-04-26T21:32:43.846138+0000","last_became_peered":"2024-04-26T21:32:43.846138+0000","last_unstale":"2024-04-26T21:32:48.894431+0000","last_undegraded":"2024-04-26T21:32:48.894431+0000","last_fullsized":"2024-04-26T21:32:48.894431+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:28:40.257244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:46.094810+0000","last_change":"2024-04-26T21:32:44.872228+0000","last_active":"2024-04-26T21:32:46.094810+0000","last_peered":"2024-04-26T21:32:46.094810+0000","last_clean":"2024-04-26T21:32:46.094810+0000","last_became_active":"2024-04-26T21:32:44.871818+0000","last_became_peered":"2024-04-26T21:32:44.871818+0000","last_unstale":"2024-04-26T21:32:46.094810+0000","last_undegraded":"2024-04-26T21:32:46.094810+0000","last_fullsized":"2024-04-26T21:32:46.094810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:52:32.520802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:48.894849+0000","last_change":"2024-04-26T21:32:43.846756+0000","last_active":"2024-04-26T21:32:48.894849+0000","last_peered":"2024-04-26T21:32:48.894849+0000","last_clean":"2024-04-26T21:32:48.894849+0000","last_became_active":"2024-04-26T21:32:43.846477+0000","last_became_peered":"2024-04-26T21:32:43.846477+0000","last_unstale":"2024-04-26T21:32:48.894849+0000","last_undegraded":"2024-04-26T21:32:48.894849+0000","last_fullsized":"2024-04-26T21:32:48.894849+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:33:27.859844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904693+0000","last_change":"2024-04-26T21:32:44.859853+0000","last_active":"2024-04-26T21:32:45.904693+0000","last_peered":"2024-04-26T21:32:45.904693+0000","last_clean":"2024-04-26T21:32:45.904693+0000","last_became_active":"2024-04-26T21:32:44.859521+0000","last_became_peered":"2024-04-26T21:32:44.859521+0000","last_unstale":"2024-04-26T21:32:45.904693+0000","last_undegraded":"2024-04-26T21:32:45.904693+0000","last_fullsized":"2024-04-26T21:32:45.904693+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:48:54.955913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:49.889630+0000","last_change":"2024-04-26T21:32:43.840795+0000","last_active":"2024-04-26T21:32:49.889630+0000","last_peered":"2024-04-26T21:32:49.889630+0000","last_clean":"2024-04-26T21:32:49.889630+0000","last_became_active":"2024-04-26T21:32:43.840443+0000","last_became_peered":"2024-04-26T21:32:43.840443+0000","last_unstale":"2024-04-26T21:32:49.889630+0000","last_undegraded":"2024-04-26T21:32:49.889630+0000","last_fullsized":"2024-04-26T21:32:49.889630+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:09:48.944085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905287+0000","last_change":"2024-04-26T21:32:44.862344+0000","last_active":"2024-04-26T21:32:45.905287+0000","last_peered":"2024-04-26T21:32:45.905287+0000","last_clean":"2024-04-26T21:32:45.905287+0000","last_became_active":"2024-04-26T21:32:44.862029+0000","last_became_peered":"2024-04-26T21:32:44.862029+0000","last_unstale":"2024-04-26T21:32:45.905287+0000","last_undegraded":"2024-04-26T21:32:45.905287+0000","last_fullsized":"2024-04-26T21:32:45.905287+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:45:48.833753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904823+0000","last_change":"2024-04-26T21:32:43.843853+0000","last_active":"2024-04-26T21:32:45.904823+0000","last_peered":"2024-04-26T21:32:45.904823+0000","last_clean":"2024-04-26T21:32:45.904823+0000","last_became_active":"2024-04-26T21:32:43.843525+0000","last_became_peered":"2024-04-26T21:32:43.843525+0000","last_unstale":"2024-04-26T21:32:45.904823+0000","last_undegraded":"2024-04-26T21:32:45.904823+0000","last_fullsized":"2024-04-26T21:32:45.904823+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:11:43.222849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.851833+0000","last_change":"2024-04-26T21:32:44.869123+0000","last_active":"2024-04-26T21:32:45.851833+0000","last_peered":"2024-04-26T21:32:45.851833+0000","last_clean":"2024-04-26T21:32:45.851833+0000","last_became_active":"2024-04-26T21:32:44.868573+0000","last_became_peered":"2024-04-26T21:32:44.868573+0000","last_unstale":"2024-04-26T21:32:45.851833+0000","last_undegraded":"2024-04-26T21:32:45.851833+0000","last_fullsized":"2024-04-26T21:32:45.851833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:48:54.447157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:46.095005+0000","last_change":"2024-04-26T21:32:43.850596+0000","last_active":"2024-04-26T21:32:46.095005+0000","last_peered":"2024-04-26T21:32:46.095005+0000","last_clean":"2024-04-26T21:32:46.095005+0000","last_became_active":"2024-04-26T21:32:43.850218+0000","last_became_peered":"2024-04-26T21:32:43.850218+0000","last_unstale":"2024-04-26T21:32:46.095005+0000","last_undegraded":"2024-04-26T21:32:46.095005+0000","last_fullsized":"2024-04-26T21:32:46.095005+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:34.068783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.851635+0000","last_change":"2024-04-26T21:32:44.863450+0000","last_active":"2024-04-26T21:32:45.851635+0000","last_peered":"2024-04-26T21:32:45.851635+0000","last_clean":"2024-04-26T21:32:45.851635+0000","last_became_active":"2024-04-26T21:32:44.863089+0000","last_became_peered":"2024-04-26T21:32:44.863089+0000","last_unstale":"2024-04-26T21:32:45.851635+0000","last_undegraded":"2024-04-26T21:32:45.851635+0000","last_fullsized":"2024-04-26T21:32:45.851635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:04:51.821073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905893+0000","last_change":"2024-04-26T21:32:44.855575+0000","last_active":"2024-04-26T21:32:45.905893+0000","last_peered":"2024-04-26T21:32:45.905893+0000","last_clean":"2024-04-26T21:32:45.905893+0000","last_became_active":"2024-04-26T21:32:44.855258+0000","last_became_peered":"2024-04-26T21:32:44.855258+0000","last_unstale":"2024-04-26T21:32:45.905893+0000","last_undegraded":"2024-04-26T21:32:45.905893+0000","last_fullsized":"2024-04-26T21:32:45.905893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:41:07.559095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:48.894928+0000","last_change":"2024-04-26T21:32:43.850713+0000","last_active":"2024-04-26T21:32:48.894928+0000","last_peered":"2024-04-26T21:32:48.894928+0000","last_clean":"2024-04-26T21:32:48.894928+0000","last_became_active":"2024-04-26T21:32:43.850406+0000","last_became_peered":"2024-04-26T21:32:43.850406+0000","last_unstale":"2024-04-26T21:32:48.894928+0000","last_undegraded":"2024-04-26T21:32:48.894928+0000","last_fullsized":"2024-04-26T21:32:48.894928+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:11:02.682963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905520+0000","last_change":"2024-04-26T21:32:44.868317+0000","last_active":"2024-04-26T21:32:45.905520+0000","last_peered":"2024-04-26T21:32:45.905520+0000","last_clean":"2024-04-26T21:32:45.905520+0000","last_became_active":"2024-04-26T21:32:44.864571+0000","last_became_peered":"2024-04-26T21:32:44.864571+0000","last_unstale":"2024-04-26T21:32:45.905520+0000","last_undegraded":"2024-04-26T21:32:45.905520+0000","last_fullsized":"2024-04-26T21:32:45.905520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:24:50.476949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:47.885709+0000","last_change":"2024-04-26T21:32:43.833773+0000","last_active":"2024-04-26T21:32:47.885709+0000","last_peered":"2024-04-26T21:32:47.885709+0000","last_clean":"2024-04-26T21:32:47.885709+0000","last_became_active":"2024-04-26T21:32:43.833338+0000","last_became_peered":"2024-04-26T21:32:43.833338+0000","last_unstale":"2024-04-26T21:32:47.885709+0000","last_undegraded":"2024-04-26T21:32:47.885709+0000","last_fullsized":"2024-04-26T21:32:47.885709+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:15.382494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":336,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:46.094775+0000","last_change":"2024-04-26T21:31:27.930370+0000","last_active":"2024-04-26T21:32:46.094775+0000","last_peered":"2024-04-26T21:32:46.094775+0000","last_clean":"2024-04-26T21:32:46.094775+0000","last_became_active":"2024-04-26T21:31:27.929859+0000","last_became_peered":"2024-04-26T21:31:27.929859+0000","last_unstale":"2024-04-26T21:32:46.094775+0000","last_undegraded":"2024-04-26T21:32:46.094775+0000","last_fullsized":"2024-04-26T21:32:46.094775+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:58.528699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:58.528699+0000","last_clean_scrub_stamp":"2024-04-26T21:27:58.528699+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:00:55.701865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.850110+0000","last_change":"2024-04-26T21:32:44.858891+0000","last_active":"2024-04-26T21:32:45.850110+0000","last_peered":"2024-04-26T21:32:45.850110+0000","last_clean":"2024-04-26T21:32:45.850110+0000","last_became_active":"2024-04-26T21:32:44.858406+0000","last_became_peered":"2024-04-26T21:32:44.858406+0000","last_unstale":"2024-04-26T21:32:45.850110+0000","last_undegraded":"2024-04-26T21:32:45.850110+0000","last_fullsized":"2024-04-26T21:32:45.850110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:40:13.084352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:48.893527+0000","last_change":"2024-04-26T21:32:43.849442+0000","last_active":"2024-04-26T21:32:48.893527+0000","last_peered":"2024-04-26T21:32:48.893527+0000","last_clean":"2024-04-26T21:32:48.893527+0000","last_became_active":"2024-04-26T21:32:43.847951+0000","last_became_peered":"2024-04-26T21:32:43.847951+0000","last_unstale":"2024-04-26T21:32:48.893527+0000","last_undegraded":"2024-04-26T21:32:48.893527+0000","last_fullsized":"2024-04-26T21:32:48.893527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:55:00.025931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:47.884695+0000","last_change":"2024-04-26T21:32:43.846736+0000","last_active":"2024-04-26T21:32:47.884695+0000","last_peered":"2024-04-26T21:32:47.884695+0000","last_clean":"2024-04-26T21:32:47.884695+0000","last_became_active":"2024-04-26T21:32:43.846451+0000","last_became_peered":"2024-04-26T21:32:43.846451+0000","last_unstale":"2024-04-26T21:32:47.884695+0000","last_undegraded":"2024-04-26T21:32:47.884695+0000","last_fullsized":"2024-04-26T21:32:47.884695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:38:46.301597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.850352+0000","last_change":"2024-04-26T21:32:44.856817+0000","last_active":"2024-04-26T21:32:45.850352+0000","last_peered":"2024-04-26T21:32:45.850352+0000","last_clean":"2024-04-26T21:32:45.850352+0000","last_became_active":"2024-04-26T21:32:44.856489+0000","last_became_peered":"2024-04-26T21:32:44.856489+0000","last_unstale":"2024-04-26T21:32:45.850352+0000","last_undegraded":"2024-04-26T21:32:45.850352+0000","last_fullsized":"2024-04-26T21:32:45.850352+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:13:33.851351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.964813+0000","last_change":"2024-04-26T21:32:43.838974+0000","last_active":"2024-04-26T21:32:45.964813+0000","last_peered":"2024-04-26T21:32:45.964813+0000","last_clean":"2024-04-26T21:32:45.964813+0000","last_became_active":"2024-04-26T21:32:43.838661+0000","last_became_peered":"2024-04-26T21:32:43.838661+0000","last_unstale":"2024-04-26T21:32:45.964813+0000","last_undegraded":"2024-04-26T21:32:45.964813+0000","last_fullsized":"2024-04-26T21:32:45.964813+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:44:29.536559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905816+0000","last_change":"2024-04-26T21:32:44.872716+0000","last_active":"2024-04-26T21:32:45.905816+0000","last_peered":"2024-04-26T21:32:45.905816+0000","last_clean":"2024-04-26T21:32:45.905816+0000","last_became_active":"2024-04-26T21:32:44.872348+0000","last_became_peered":"2024-04-26T21:32:44.872348+0000","last_unstale":"2024-04-26T21:32:45.905816+0000","last_undegraded":"2024-04-26T21:32:45.905816+0000","last_fullsized":"2024-04-26T21:32:45.905816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:17:56.890488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:47.884204+0000","last_change":"2024-04-26T21:32:43.853138+0000","last_active":"2024-04-26T21:32:47.884204+0000","last_peered":"2024-04-26T21:32:47.884204+0000","last_clean":"2024-04-26T21:32:47.884204+0000","last_became_active":"2024-04-26T21:32:43.852803+0000","last_became_peered":"2024-04-26T21:32:43.852803+0000","last_unstale":"2024-04-26T21:32:47.884204+0000","last_undegraded":"2024-04-26T21:32:47.884204+0000","last_fullsized":"2024-04-26T21:32:47.884204+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:59:04.790442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904591+0000","last_change":"2024-04-26T21:32:44.867559+0000","last_active":"2024-04-26T21:32:45.904591+0000","last_peered":"2024-04-26T21:32:45.904591+0000","last_clean":"2024-04-26T21:32:45.904591+0000","last_became_active":"2024-04-26T21:32:44.867229+0000","last_became_peered":"2024-04-26T21:32:44.867229+0000","last_unstale":"2024-04-26T21:32:45.904591+0000","last_undegraded":"2024-04-26T21:32:45.904591+0000","last_fullsized":"2024-04-26T21:32:45.904591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:09:09.052753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:47.885134+0000","last_change":"2024-04-26T21:32:43.844761+0000","last_active":"2024-04-26T21:32:47.885134+0000","last_peered":"2024-04-26T21:32:47.885134+0000","last_clean":"2024-04-26T21:32:47.885134+0000","last_became_active":"2024-04-26T21:32:43.844257+0000","last_became_peered":"2024-04-26T21:32:43.844257+0000","last_unstale":"2024-04-26T21:32:47.885134+0000","last_undegraded":"2024-04-26T21:32:47.885134+0000","last_fullsized":"2024-04-26T21:32:47.885134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:06:55.560467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904670+0000","last_change":"2024-04-26T21:32:44.856662+0000","last_active":"2024-04-26T21:32:45.904670+0000","last_peered":"2024-04-26T21:32:45.904670+0000","last_clean":"2024-04-26T21:32:45.904670+0000","last_became_active":"2024-04-26T21:32:44.856297+0000","last_became_peered":"2024-04-26T21:32:44.856297+0000","last_unstale":"2024-04-26T21:32:45.904670+0000","last_undegraded":"2024-04-26T21:32:45.904670+0000","last_fullsized":"2024-04-26T21:32:45.904670+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:48:55.314959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:47.884729+0000","last_change":"2024-04-26T21:32:43.848710+0000","last_active":"2024-04-26T21:32:47.884729+0000","last_peered":"2024-04-26T21:32:47.884729+0000","last_clean":"2024-04-26T21:32:47.884729+0000","last_became_active":"2024-04-26T21:32:43.848379+0000","last_became_peered":"2024-04-26T21:32:43.848379+0000","last_unstale":"2024-04-26T21:32:47.884729+0000","last_undegraded":"2024-04-26T21:32:47.884729+0000","last_fullsized":"2024-04-26T21:32:47.884729+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:26:44.233330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905408+0000","last_change":"2024-04-26T21:32:44.858327+0000","last_active":"2024-04-26T21:32:45.905408+0000","last_peered":"2024-04-26T21:32:45.905408+0000","last_clean":"2024-04-26T21:32:45.905408+0000","last_became_active":"2024-04-26T21:32:44.858023+0000","last_became_peered":"2024-04-26T21:32:44.858023+0000","last_unstale":"2024-04-26T21:32:45.905408+0000","last_undegraded":"2024-04-26T21:32:45.905408+0000","last_fullsized":"2024-04-26T21:32:45.905408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:30.588063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.851623+0000","last_change":"2024-04-26T21:32:43.839304+0000","last_active":"2024-04-26T21:32:45.851623+0000","last_peered":"2024-04-26T21:32:45.851623+0000","last_clean":"2024-04-26T21:32:45.851623+0000","last_became_active":"2024-04-26T21:32:43.838875+0000","last_became_peered":"2024-04-26T21:32:43.838875+0000","last_unstale":"2024-04-26T21:32:45.851623+0000","last_undegraded":"2024-04-26T21:32:45.851623+0000","last_fullsized":"2024-04-26T21:32:45.851623+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:07:48.761227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905769+0000","last_change":"2024-04-26T21:32:44.867088+0000","last_active":"2024-04-26T21:32:45.905769+0000","last_peered":"2024-04-26T21:32:45.905769+0000","last_clean":"2024-04-26T21:32:45.905769+0000","last_became_active":"2024-04-26T21:32:44.863623+0000","last_became_peered":"2024-04-26T21:32:44.863623+0000","last_unstale":"2024-04-26T21:32:45.905769+0000","last_undegraded":"2024-04-26T21:32:45.905769+0000","last_fullsized":"2024-04-26T21:32:45.905769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:42:21.107362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:47.884500+0000","last_change":"2024-04-26T21:32:43.844740+0000","last_active":"2024-04-26T21:32:47.884500+0000","last_peered":"2024-04-26T21:32:47.884500+0000","last_clean":"2024-04-26T21:32:47.884500+0000","last_became_active":"2024-04-26T21:32:43.844453+0000","last_became_peered":"2024-04-26T21:32:43.844453+0000","last_unstale":"2024-04-26T21:32:47.884500+0000","last_undegraded":"2024-04-26T21:32:47.884500+0000","last_fullsized":"2024-04-26T21:32:47.884500+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:34:02.264468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:46.095377+0000","last_change":"2024-04-26T21:32:44.863365+0000","last_active":"2024-04-26T21:32:46.095377+0000","last_peered":"2024-04-26T21:32:46.095377+0000","last_clean":"2024-04-26T21:32:46.095377+0000","last_became_active":"2024-04-26T21:32:44.863012+0000","last_became_peered":"2024-04-26T21:32:44.863012+0000","last_unstale":"2024-04-26T21:32:46.095377+0000","last_undegraded":"2024-04-26T21:32:46.095377+0000","last_fullsized":"2024-04-26T21:32:46.095377+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:09:50.023773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.852062+0000","last_change":"2024-04-26T21:32:43.853507+0000","last_active":"2024-04-26T21:32:45.852062+0000","last_peered":"2024-04-26T21:32:45.852062+0000","last_clean":"2024-04-26T21:32:45.852062+0000","last_became_active":"2024-04-26T21:32:43.852680+0000","last_became_peered":"2024-04-26T21:32:43.852680+0000","last_unstale":"2024-04-26T21:32:45.852062+0000","last_undegraded":"2024-04-26T21:32:45.852062+0000","last_fullsized":"2024-04-26T21:32:45.852062+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:46:51.836736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:46.095361+0000","last_change":"2024-04-26T21:32:44.865158+0000","last_active":"2024-04-26T21:32:46.095361+0000","last_peered":"2024-04-26T21:32:46.095361+0000","last_clean":"2024-04-26T21:32:46.095361+0000","last_became_active":"2024-04-26T21:32:44.864182+0000","last_became_peered":"2024-04-26T21:32:44.864182+0000","last_unstale":"2024-04-26T21:32:46.095361+0000","last_undegraded":"2024-04-26T21:32:46.095361+0000","last_fullsized":"2024-04-26T21:32:46.095361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:55.918623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905744+0000","last_change":"2024-04-26T21:32:43.842913+0000","last_active":"2024-04-26T21:32:45.905744+0000","last_peered":"2024-04-26T21:32:45.905744+0000","last_clean":"2024-04-26T21:32:45.905744+0000","last_became_active":"2024-04-26T21:32:43.842599+0000","last_became_peered":"2024-04-26T21:32:43.842599+0000","last_unstale":"2024-04-26T21:32:45.905744+0000","last_undegraded":"2024-04-26T21:32:45.905744+0000","last_fullsized":"2024-04-26T21:32:45.905744+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:46.583398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:46.095177+0000","last_change":"2024-04-26T21:32:44.864239+0000","last_active":"2024-04-26T21:32:46.095177+0000","last_peered":"2024-04-26T21:32:46.095177+0000","last_clean":"2024-04-26T21:32:46.095177+0000","last_became_active":"2024-04-26T21:32:44.863862+0000","last_became_peered":"2024-04-26T21:32:44.863862+0000","last_unstale":"2024-04-26T21:32:46.095177+0000","last_undegraded":"2024-04-26T21:32:46.095177+0000","last_fullsized":"2024-04-26T21:32:46.095177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:32:16.958044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.850363+0000","last_change":"2024-04-26T21:32:43.849768+0000","last_active":"2024-04-26T21:32:45.850363+0000","last_peered":"2024-04-26T21:32:45.850363+0000","last_clean":"2024-04-26T21:32:45.850363+0000","last_became_active":"2024-04-26T21:32:43.849424+0000","last_became_peered":"2024-04-26T21:32:43.849424+0000","last_unstale":"2024-04-26T21:32:45.850363+0000","last_undegraded":"2024-04-26T21:32:45.850363+0000","last_fullsized":"2024-04-26T21:32:45.850363+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:46:23.231819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:46.095209+0000","last_change":"2024-04-26T21:32:44.862597+0000","last_active":"2024-04-26T21:32:46.095209+0000","last_peered":"2024-04-26T21:32:46.095209+0000","last_clean":"2024-04-26T21:32:46.095209+0000","last_became_active":"2024-04-26T21:32:44.862259+0000","last_became_peered":"2024-04-26T21:32:44.862259+0000","last_unstale":"2024-04-26T21:32:46.095209+0000","last_undegraded":"2024-04-26T21:32:46.095209+0000","last_fullsized":"2024-04-26T21:32:46.095209+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:19.759329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905522+0000","last_change":"2024-04-26T21:32:44.855196+0000","last_active":"2024-04-26T21:32:45.905522+0000","last_peered":"2024-04-26T21:32:45.905522+0000","last_clean":"2024-04-26T21:32:45.905522+0000","last_became_active":"2024-04-26T21:32:44.854774+0000","last_became_peered":"2024-04-26T21:32:44.854774+0000","last_unstale":"2024-04-26T21:32:45.905522+0000","last_undegraded":"2024-04-26T21:32:45.905522+0000","last_fullsized":"2024-04-26T21:32:45.905522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:46:18.370983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.851774+0000","last_change":"2024-04-26T21:32:44.868115+0000","last_active":"2024-04-26T21:32:45.851774+0000","last_peered":"2024-04-26T21:32:45.851774+0000","last_clean":"2024-04-26T21:32:45.851774+0000","last_became_active":"2024-04-26T21:32:44.867767+0000","last_became_peered":"2024-04-26T21:32:44.867767+0000","last_unstale":"2024-04-26T21:32:45.851774+0000","last_undegraded":"2024-04-26T21:32:45.851774+0000","last_fullsized":"2024-04-26T21:32:45.851774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:30:22.130786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.851815+0000","last_change":"2024-04-26T21:32:44.864633+0000","last_active":"2024-04-26T21:32:45.851815+0000","last_peered":"2024-04-26T21:32:45.851815+0000","last_clean":"2024-04-26T21:32:45.851815+0000","last_became_active":"2024-04-26T21:32:44.864245+0000","last_became_peered":"2024-04-26T21:32:44.864245+0000","last_unstale":"2024-04-26T21:32:45.851815+0000","last_undegraded":"2024-04-26T21:32:45.851815+0000","last_fullsized":"2024-04-26T21:32:45.851815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:10:20.315396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904498+0000","last_change":"2024-04-26T21:32:44.858122+0000","last_active":"2024-04-26T21:32:45.904498+0000","last_peered":"2024-04-26T21:32:45.904498+0000","last_clean":"2024-04-26T21:32:45.904498+0000","last_became_active":"2024-04-26T21:32:44.857513+0000","last_became_peered":"2024-04-26T21:32:44.857513+0000","last_unstale":"2024-04-26T21:32:45.904498+0000","last_undegraded":"2024-04-26T21:32:45.904498+0000","last_fullsized":"2024-04-26T21:32:45.904498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:50:40.725215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.850435+0000","last_change":"2024-04-26T21:32:44.858684+0000","last_active":"2024-04-26T21:32:45.850435+0000","last_peered":"2024-04-26T21:32:45.850435+0000","last_clean":"2024-04-26T21:32:45.850435+0000","last_became_active":"2024-04-26T21:32:44.858298+0000","last_became_peered":"2024-04-26T21:32:44.858298+0000","last_unstale":"2024-04-26T21:32:45.850435+0000","last_undegraded":"2024-04-26T21:32:45.850435+0000","last_fullsized":"2024-04-26T21:32:45.850435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:52:00.512883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905022+0000","last_change":"2024-04-26T21:32:44.852073+0000","last_active":"2024-04-26T21:32:45.905022+0000","last_peered":"2024-04-26T21:32:45.905022+0000","last_clean":"2024-04-26T21:32:45.905022+0000","last_became_active":"2024-04-26T21:32:44.851717+0000","last_became_peered":"2024-04-26T21:32:44.851717+0000","last_unstale":"2024-04-26T21:32:45.905022+0000","last_undegraded":"2024-04-26T21:32:45.905022+0000","last_fullsized":"2024-04-26T21:32:45.905022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:27:37.599933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.852196+0000","last_change":"2024-04-26T21:32:44.859503+0000","last_active":"2024-04-26T21:32:45.852196+0000","last_peered":"2024-04-26T21:32:45.852196+0000","last_clean":"2024-04-26T21:32:45.852196+0000","last_became_active":"2024-04-26T21:32:44.859009+0000","last_became_peered":"2024-04-26T21:32:44.859009+0000","last_unstale":"2024-04-26T21:32:45.852196+0000","last_undegraded":"2024-04-26T21:32:45.852196+0000","last_fullsized":"2024-04-26T21:32:45.852196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:00:26.307567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905758+0000","last_change":"2024-04-26T21:32:44.875092+0000","last_active":"2024-04-26T21:32:45.905758+0000","last_peered":"2024-04-26T21:32:45.905758+0000","last_clean":"2024-04-26T21:32:45.905758+0000","last_became_active":"2024-04-26T21:32:44.874819+0000","last_became_peered":"2024-04-26T21:32:44.874819+0000","last_unstale":"2024-04-26T21:32:45.905758+0000","last_undegraded":"2024-04-26T21:32:45.905758+0000","last_fullsized":"2024-04-26T21:32:45.905758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:48:34.178656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905891+0000","last_change":"2024-04-26T21:32:44.857666+0000","last_active":"2024-04-26T21:32:45.905891+0000","last_peered":"2024-04-26T21:32:45.905891+0000","last_clean":"2024-04-26T21:32:45.905891+0000","last_became_active":"2024-04-26T21:32:44.857342+0000","last_became_peered":"2024-04-26T21:32:44.857342+0000","last_unstale":"2024-04-26T21:32:45.905891+0000","last_undegraded":"2024-04-26T21:32:45.905891+0000","last_fullsized":"2024-04-26T21:32:45.905891+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:05:15.139919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.905141+0000","last_change":"2024-04-26T21:32:44.868571+0000","last_active":"2024-04-26T21:32:45.905141+0000","last_peered":"2024-04-26T21:32:45.905141+0000","last_clean":"2024-04-26T21:32:45.905141+0000","last_became_active":"2024-04-26T21:32:44.868191+0000","last_became_peered":"2024-04-26T21:32:44.868191+0000","last_unstale":"2024-04-26T21:32:45.905141+0000","last_undegraded":"2024-04-26T21:32:45.905141+0000","last_fullsized":"2024-04-26T21:32:45.905141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:35.329277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904913+0000","last_change":"2024-04-26T21:32:44.867823+0000","last_active":"2024-04-26T21:32:45.904913+0000","last_peered":"2024-04-26T21:32:45.904913+0000","last_clean":"2024-04-26T21:32:45.904913+0000","last_became_active":"2024-04-26T21:32:44.867496+0000","last_became_peered":"2024-04-26T21:32:44.867496+0000","last_unstale":"2024-04-26T21:32:45.904913+0000","last_undegraded":"2024-04-26T21:32:45.904913+0000","last_fullsized":"2024-04-26T21:32:45.904913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:41:47.484332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.850514+0000","last_change":"2024-04-26T21:32:44.864720+0000","last_active":"2024-04-26T21:32:45.850514+0000","last_peered":"2024-04-26T21:32:45.850514+0000","last_clean":"2024-04-26T21:32:45.850514+0000","last_became_active":"2024-04-26T21:32:44.864395+0000","last_became_peered":"2024-04-26T21:32:44.864395+0000","last_unstale":"2024-04-26T21:32:45.850514+0000","last_undegraded":"2024-04-26T21:32:45.850514+0000","last_fullsized":"2024-04-26T21:32:45.850514+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:25.416249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-26T21:32:45.904758+0000","last_change":"2024-04-26T21:32:44.862415+0000","last_active":"2024-04-26T21:32:45.904758+0000","last_peered":"2024-04-26T21:32:45.904758+0000","last_clean":"2024-04-26T21:32:45.904758+0000","last_became_active":"2024-04-26T21:32:44.862141+0000","last_became_peered":"2024-04-26T21:32:44.862141+0000","last_unstale":"2024-04-26T21:32:45.904758+0000","last_undegraded":"2024-04-26T21:32:45.904758+0000","last_fullsized":"2024-04-26T21:32:45.904758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:52:55.919407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280902,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":105,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":21406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93,"ondisk_log_size":93,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383695,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":917504,"data_stored":439103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645331,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":1032079,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808856,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":917504,"data_stored":439103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070494,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":933888,"data_stored":439259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332131,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":1031463,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495656,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":438521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40400,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704752,"statfs":{"total":95995035648,"available":95953666048,"internally_reserved":0,"allocated":925696,"data_stored":440133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":439999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":929792,"data_stored":440167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36240,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708912,"statfs":{"total":95995035648,"available":95957925888,"internally_reserved":0,"allocated":925696,"data_stored":438715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920819,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36240,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708912,"statfs":{"total":95995035648,"available":95957925888,"internally_reserved":0,"allocated":925696,"data_stored":439165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757294,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":991232,"data_stored":1030365,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34089,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34089,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34089,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":8212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T21:32:53.087 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-26T21:32:53.088 INFO:tasks.cephadm:Setting up client nodes... 2024-04-26T21:32:53.088 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-26T21:32:53.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:52 smithi142 ceph-mon[32944]: pgmap v296: 97 pgs: 97 active+clean; 581 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 4.6 KiB/s wr, 13 op/s 2024-04-26T21:32:53.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:52 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3166015083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:53.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:52 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/1925659073' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:32:53.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:52 smithi142 ceph-mon[32944]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:32:53.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:52 smithi050 ceph-mon[32753]: pgmap v296: 97 pgs: 97 active+clean; 581 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 4.6 KiB/s wr, 13 op/s 2024-04-26T21:32:53.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:52 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3166015083' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:32:53.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:52 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/1925659073' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:32:53.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:52 smithi050 ceph-mon[32753]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:32:53.372 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:32:54.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:53 smithi046 ceph-mon[29201]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-26T21:32:54.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:53 smithi046 ceph-mon[29201]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:32:54.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:53 smithi142 ceph-mon[32944]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-26T21:32:54.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:53 smithi142 ceph-mon[32944]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:32:54.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:53 smithi050 ceph-mon[32753]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-26T21:32:54.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:53 smithi050 ceph-mon[32753]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:32:55.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.172+0000 7fad3c811700 1 -- 172.21.15.46:0/1941693357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad381035a0 msgr2=0x7fad38103a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:55.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.172+0000 7fad3c811700 1 --2- 172.21.15.46:0/1941693357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad381035a0 0x7fad38103a00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fad2c0099c0 tx=0x7fad2c009cd0 comp rx=0 tx=0).stop 2024-04-26T21:32:55.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.172+0000 7fad3c811700 1 -- 172.21.15.46:0/1941693357 shutdown_connections 2024-04-26T21:32:55.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.172+0000 7fad3c811700 1 --2- 172.21.15.46:0/1941693357 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad38103f40 0x7fad3810a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:55.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.172+0000 7fad3c811700 1 --2- 172.21.15.46:0/1941693357 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad381023a0 0x7fad381027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:55.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.172+0000 7fad3c811700 1 --2- 172.21.15.46:0/1941693357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad381035a0 0x7fad38103a00 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:55.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.172+0000 7fad3c811700 1 -- 172.21.15.46:0/1941693357 >> 172.21.15.46:0/1941693357 conn(0x7fad380fd940 msgr2=0x7fad380ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:55.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.172+0000 7fad3c811700 1 -- 172.21.15.46:0/1941693357 shutdown_connections 2024-04-26T21:32:55.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.172+0000 7fad3c811700 1 -- 172.21.15.46:0/1941693357 wait complete. 2024-04-26T21:32:55.175 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:54 smithi046 ceph-mon[29201]: pgmap v297: 97 pgs: 97 active+clean; 582 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 901 B/s rd, 4.4 KiB/s wr, 15 op/s 2024-04-26T21:32:55.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.173+0000 7fad3c811700 1 Processor -- start 2024-04-26T21:32:55.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.173+0000 7fad3c811700 1 -- start start 2024-04-26T21:32:55.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.173+0000 7fad3c811700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad381023a0 0x7fad38199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:55.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.173+0000 7fad3c811700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad381035a0 0x7fad38199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:55.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.173+0000 7fad3c811700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad38103f40 0x7fad381a0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:55.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.173+0000 7fad3c811700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fad3810de40 con 0x7fad381023a0 2024-04-26T21:32:55.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.173+0000 7fad3c811700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fad3810dcc0 con 0x7fad381035a0 2024-04-26T21:32:55.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.173+0000 7fad3c811700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fad3810dfc0 con 0x7fad38103f40 2024-04-26T21:32:55.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.173+0000 7fad3659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad381023a0 0x7fad38199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:55.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.173+0000 7fad3659c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad381023a0 0x7fad38199510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49594/0 (socket says 172.21.15.46:49594) 2024-04-26T21:32:55.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.174+0000 7fad3659c700 1 -- 172.21.15.46:0/662030845 learned_addr learned my addr 172.21.15.46:0/662030845 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:55.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.174+0000 7fad35d9b700 1 --2- 172.21.15.46:0/662030845 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad381035a0 0x7fad38199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:55.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.174+0000 7fad36d9d700 1 --2- 172.21.15.46:0/662030845 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad38103f40 0x7fad381a0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:55.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.174+0000 7fad3659c700 1 -- 172.21.15.46:0/662030845 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad38103f40 msgr2=0x7fad381a0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:55.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.174+0000 7fad3659c700 1 --2- 172.21.15.46:0/662030845 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad38103f40 0x7fad381a0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:55.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.174+0000 7fad3659c700 1 -- 172.21.15.46:0/662030845 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad381035a0 msgr2=0x7fad38199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:55.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.174+0000 7fad3659c700 1 --2- 172.21.15.46:0/662030845 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad381035a0 0x7fad38199a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:55.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.174+0000 7fad3659c700 1 -- 172.21.15.46:0/662030845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad381a1170 con 0x7fad381023a0 2024-04-26T21:32:55.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.174+0000 7fad3659c700 1 --2- 172.21.15.46:0/662030845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad381023a0 0x7fad38199510 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fad2800d9b0 tx=0x7fad2800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:55.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.174+0000 7fad277fe700 1 -- 172.21.15.46:0/662030845 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad28016070 con 0x7fad381023a0 2024-04-26T21:32:55.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.174+0000 7fad277fe700 1 -- 172.21.15.46:0/662030845 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad2800be00 con 0x7fad381023a0 2024-04-26T21:32:55.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.174+0000 7fad3c811700 1 -- 172.21.15.46:0/662030845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad381a1460 con 0x7fad381023a0 2024-04-26T21:32:55.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.174+0000 7fad277fe700 1 -- 172.21.15.46:0/662030845 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad28010e30 con 0x7fad381023a0 2024-04-26T21:32:55.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.174+0000 7fad3c811700 1 -- 172.21.15.46:0/662030845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad381a19f0 con 0x7fad381023a0 2024-04-26T21:32:55.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.176+0000 7fad3c811700 1 -- 172.21.15.46:0/662030845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad38051960 con 0x7fad381023a0 2024-04-26T21:32:55.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.176+0000 7fad277fe700 1 -- 172.21.15.46:0/662030845 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fad28019070 con 0x7fad381023a0 2024-04-26T21:32:55.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.177+0000 7fad277fe700 1 --2- 172.21.15.46:0/662030845 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fad200a3e00 0x7fad200a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:55.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.177+0000 7fad35d9b700 1 --2- 172.21.15.46:0/662030845 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fad200a3e00 0x7fad200a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:55.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.177+0000 7fad277fe700 1 -- 172.21.15.46:0/662030845 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fad280d01f0 con 0x7fad381023a0 2024-04-26T21:32:55.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.178+0000 7fad35d9b700 1 --2- 172.21.15.46:0/662030845 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fad200a3e00 0x7fad200a62c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fad2c009990 tx=0x7fad2c0096b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:55.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.182+0000 7fad277fe700 1 -- 172.21.15.46:0/662030845 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fad28020020 con 0x7fad381023a0 2024-04-26T21:32:55.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:54 smithi142 ceph-mon[32944]: pgmap v297: 97 pgs: 97 active+clean; 582 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 901 B/s rd, 4.4 KiB/s wr, 15 op/s 2024-04-26T21:32:55.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:54 smithi050 ceph-mon[32753]: pgmap v297: 97 pgs: 97 active+clean; 582 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 901 B/s rd, 4.4 KiB/s wr, 15 op/s 2024-04-26T21:32:55.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.473+0000 7fad3c811700 1 -- 172.21.15.46:0/662030845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fad381a1e20 con 0x7fad381023a0 2024-04-26T21:32:55.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.475+0000 7fad277fe700 1 -- 172.21.15.46:0/662030845 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fad2809cc20 con 0x7fad381023a0 2024-04-26T21:32:55.476 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2024-04-26T21:32:55.476 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQCHHSxmb/xKHBAAFS2J51qQxwZTLVO+28s/Kw== 2024-04-26T21:32:55.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.477+0000 7fad3c811700 1 -- 172.21.15.46:0/662030845 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fad200a3e00 msgr2=0x7fad200a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:55.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.477+0000 7fad3c811700 1 --2- 172.21.15.46:0/662030845 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fad200a3e00 0x7fad200a62c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fad2c009990 tx=0x7fad2c0096b0 comp rx=0 tx=0).stop 2024-04-26T21:32:55.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.477+0000 7fad3c811700 1 -- 172.21.15.46:0/662030845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad381023a0 msgr2=0x7fad38199510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:55.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.477+0000 7fad3c811700 1 --2- 172.21.15.46:0/662030845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad381023a0 0x7fad38199510 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fad2800d9b0 tx=0x7fad2800dcc0 comp rx=0 tx=0).stop 2024-04-26T21:32:55.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.477+0000 7fad3c811700 1 -- 172.21.15.46:0/662030845 shutdown_connections 2024-04-26T21:32:55.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.477+0000 7fad3c811700 1 --2- 172.21.15.46:0/662030845 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fad200a3e00 0x7fad200a62c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:55.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.477+0000 7fad3c811700 1 --2- 172.21.15.46:0/662030845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad381023a0 0x7fad38199510 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:55.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.477+0000 7fad3c811700 1 --2- 172.21.15.46:0/662030845 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fad38103f40 0x7fad381a0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:55.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.477+0000 7fad3c811700 1 --2- 172.21.15.46:0/662030845 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fad381035a0 0x7fad38199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:55.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.477+0000 7fad3c811700 1 -- 172.21.15.46:0/662030845 >> 172.21.15.46:0/662030845 conn(0x7fad380fd940 msgr2=0x7fad380ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:55.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.477+0000 7fad3c811700 1 -- 172.21.15.46:0/662030845 shutdown_connections 2024-04-26T21:32:55.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:55.477+0000 7fad3c811700 1 -- 172.21.15.46:0/662030845 wait complete. 2024-04-26T21:32:56.199 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:55 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/662030845' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T21:32:56.199 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:55 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/662030845' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T21:32:56.228 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-26T21:32:56.228 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-26T21:32:56.229 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-26T21:32:56.266 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-26T21:32:56.267 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-26T21:32:56.267 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-26T21:32:56.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:55 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/662030845' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T21:32:56.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:55 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/662030845' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T21:32:56.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:55 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/662030845' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T21:32:56.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:55 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/662030845' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T21:32:56.551 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:32:57.045 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:56 smithi046 ceph-mon[29201]: pgmap v298: 97 pgs: 97 active+clean; 623 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 8.6 KiB/s wr, 17 op/s 2024-04-26T21:32:57.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:56 smithi142 ceph-mon[32944]: pgmap v298: 97 pgs: 97 active+clean; 623 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 8.6 KiB/s wr, 17 op/s 2024-04-26T21:32:57.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:56 smithi050 ceph-mon[32753]: pgmap v298: 97 pgs: 97 active+clean; 623 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 8.6 KiB/s wr, 17 op/s 2024-04-26T21:32:58.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.393+0000 7f2719adc700 1 -- 172.21.15.46:0/2838834394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27141034a0 msgr2=0x7f2714103920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:58.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.393+0000 7f2719adc700 1 --2- 172.21.15.46:0/2838834394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27141034a0 0x7f2714103920 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2708009990 tx=0x7f2708009ca0 comp rx=0 tx=0).stop 2024-04-26T21:32:58.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.394+0000 7f2719adc700 1 -- 172.21.15.46:0/2838834394 shutdown_connections 2024-04-26T21:32:58.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.394+0000 7f2719adc700 1 --2- 172.21.15.46:0/2838834394 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2714103e60 0x7f27141102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:58.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.394+0000 7f2719adc700 1 --2- 172.21.15.46:0/2838834394 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f27141094e0 0x7f27141098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:58.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.394+0000 7f2719adc700 1 --2- 172.21.15.46:0/2838834394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27141034a0 0x7f2714103920 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:58.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.394+0000 7f2719adc700 1 -- 172.21.15.46:0/2838834394 >> 172.21.15.46:0/2838834394 conn(0x7f271407a590 msgr2=0x7f271407a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:58.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.394+0000 7f2719adc700 1 -- 172.21.15.46:0/2838834394 shutdown_connections 2024-04-26T21:32:58.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.394+0000 7f2719adc700 1 -- 172.21.15.46:0/2838834394 wait complete. 2024-04-26T21:32:58.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.394+0000 7f2719adc700 1 Processor -- start 2024-04-26T21:32:58.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.395+0000 7f2719adc700 1 -- start start 2024-04-26T21:32:58.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.395+0000 7f2719adc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27141034a0 0x7f27141a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:58.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.395+0000 7f2719adc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2714103e60 0x7f27141a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:58.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.395+0000 7f2719adc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f27141094e0 0x7f271419bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:58.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.395+0000 7f2719adc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f27141135e0 con 0x7f2714103e60 2024-04-26T21:32:58.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.395+0000 7f2719adc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2714113460 con 0x7f27141094e0 2024-04-26T21:32:58.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.395+0000 7f2719adc700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2714113760 con 0x7f27141034a0 2024-04-26T21:32:58.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.395+0000 7f2712ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2714103e60 0x7f27141a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:58.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.395+0000 7f2712ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2714103e60 0x7f27141a1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59776/0 (socket says 172.21.15.46:59776) 2024-04-26T21:32:58.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.395+0000 7f2712ffd700 1 -- 172.21.15.46:0/3537924864 learned_addr learned my addr 172.21.15.46:0/3537924864 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:32:58.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.395+0000 7f2713fff700 1 --2- 172.21.15.46:0/3537924864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f27141094e0 0x7f271419bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:58.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.395+0000 7f27137fe700 1 --2- 172.21.15.46:0/3537924864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27141034a0 0x7f27141a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:58.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.395+0000 7f2712ffd700 1 -- 172.21.15.46:0/3537924864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27141034a0 msgr2=0x7f27141a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:58.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.395+0000 7f2712ffd700 1 --2- 172.21.15.46:0/3537924864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27141034a0 0x7f27141a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:58.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.395+0000 7f2712ffd700 1 -- 172.21.15.46:0/3537924864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f27141094e0 msgr2=0x7f271419bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:58.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.395+0000 7f2712ffd700 1 --2- 172.21.15.46:0/3537924864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f27141094e0 0x7f271419bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:58.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.395+0000 7f2712ffd700 1 -- 172.21.15.46:0/3537924864 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f271419c260 con 0x7f2714103e60 2024-04-26T21:32:58.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.396+0000 7f2712ffd700 1 --2- 172.21.15.46:0/3537924864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2714103e60 0x7f27141a1e80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2708000c00 tx=0x7f27080047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:58.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.396+0000 7f2710ff9700 1 -- 172.21.15.46:0/3537924864 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2708027070 con 0x7f2714103e60 2024-04-26T21:32:58.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.396+0000 7f2719adc700 1 -- 172.21.15.46:0/3537924864 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f271419c4f0 con 0x7f2714103e60 2024-04-26T21:32:58.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.396+0000 7f2710ff9700 1 -- 172.21.15.46:0/3537924864 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2708003f20 con 0x7f2714103e60 2024-04-26T21:32:58.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.396+0000 7f2710ff9700 1 -- 172.21.15.46:0/3537924864 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2708022cf0 con 0x7f2714103e60 2024-04-26T21:32:58.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.396+0000 7f2719adc700 1 -- 172.21.15.46:0/3537924864 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27141a8810 con 0x7f2714103e60 2024-04-26T21:32:58.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.397+0000 7f2719adc700 1 -- 172.21.15.46:0/3537924864 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f271406a110 con 0x7f2714103e60 2024-04-26T21:32:58.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.398+0000 7f2710ff9700 1 -- 172.21.15.46:0/3537924864 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f270802a070 con 0x7f2714103e60 2024-04-26T21:32:58.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.398+0000 7f2710ff9700 1 --2- 172.21.15.46:0/3537924864 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f26fc0a3d50 0x7f26fc0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:58.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.399+0000 7f27137fe700 1 --2- 172.21.15.46:0/3537924864 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f26fc0a3d50 0x7f26fc0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:58.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.399+0000 7f2710ff9700 1 -- 172.21.15.46:0/3537924864 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f27080d1220 con 0x7f2714103e60 2024-04-26T21:32:58.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.399+0000 7f27137fe700 1 --2- 172.21.15.46:0/3537924864 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f26fc0a3d50 0x7f26fc0a6210 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2704005950 tx=0x7f27040058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:58.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.403+0000 7f2710ff9700 1 -- 172.21.15.46:0/3537924864 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2708013070 con 0x7f2714103e60 2024-04-26T21:32:58.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.669+0000 7f2719adc700 1 -- 172.21.15.46:0/3537924864 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f27141a8a40 con 0x7f2714103e60 2024-04-26T21:32:58.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.674+0000 7f2710ff9700 1 -- 172.21.15.46:0/3537924864 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264997 (secure 0 0 0) 0x7f270809dbe0 con 0x7f2714103e60 2024-04-26T21:32:58.680 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:32:58.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.677+0000 7f2719adc700 1 -- 172.21.15.46:0/3537924864 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f26fc0a3d50 msgr2=0x7f26fc0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:58.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.677+0000 7f2719adc700 1 --2- 172.21.15.46:0/3537924864 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f26fc0a3d50 0x7f26fc0a6210 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2704005950 tx=0x7f27040058e0 comp rx=0 tx=0).stop 2024-04-26T21:32:58.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.677+0000 7f2719adc700 1 -- 172.21.15.46:0/3537924864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2714103e60 msgr2=0x7f27141a1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:58.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.677+0000 7f2719adc700 1 --2- 172.21.15.46:0/3537924864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2714103e60 0x7f27141a1e80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2708000c00 tx=0x7f27080047b0 comp rx=0 tx=0).stop 2024-04-26T21:32:58.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.678+0000 7f2719adc700 1 -- 172.21.15.46:0/3537924864 shutdown_connections 2024-04-26T21:32:58.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.678+0000 7f2719adc700 1 --2- 172.21.15.46:0/3537924864 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f27141094e0 0x7f271419bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:58.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.678+0000 7f2719adc700 1 --2- 172.21.15.46:0/3537924864 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f27141034a0 0x7f27141a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:58.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.678+0000 7f2719adc700 1 --2- 172.21.15.46:0/3537924864 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f26fc0a3d50 0x7f26fc0a6210 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:58.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.678+0000 7f2719adc700 1 --2- 172.21.15.46:0/3537924864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2714103e60 0x7f27141a1e80 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:58.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.678+0000 7f2719adc700 1 -- 172.21.15.46:0/3537924864 >> 172.21.15.46:0/3537924864 conn(0x7f271407a590 msgr2=0x7f271410e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:58.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.678+0000 7f2719adc700 1 -- 172.21.15.46:0/3537924864 shutdown_connections 2024-04-26T21:32:58.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:32:58.678+0000 7f2719adc700 1 -- 172.21.15.46:0/3537924864 wait complete. 2024-04-26T21:32:59.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:58 smithi142 ceph-mon[32944]: pgmap v299: 97 pgs: 97 active+clean; 623 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 7.3 KiB/s wr, 15 op/s 2024-04-26T21:32:59.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:32:58 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3537924864' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T21:32:59.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:58 smithi046 ceph-mon[29201]: pgmap v299: 97 pgs: 97 active+clean; 623 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 7.3 KiB/s wr, 15 op/s 2024-04-26T21:32:59.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:32:58 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3537924864' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T21:32:59.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:58 smithi050 ceph-mon[32753]: pgmap v299: 97 pgs: 97 active+clean; 623 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 7.3 KiB/s wr, 15 op/s 2024-04-26T21:32:59.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:32:58 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3537924864' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T21:32:59.433 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":4004175376},{"type":"v1","addr":"172.21.15.46:6801","nonce":4004175376}]},"active_addr":"172.21.15.46:6801/4004175376","active_change":"2024-04-26T21:25:29.848503+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24119,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.46:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":3061734445}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":3408180399}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":789219039}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":3529270683}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":3665279237}]}]} 2024-04-26T21:32:59.439 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-26T21:32:59.440 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-26T21:32:59.440 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T21:32:59.687 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:33:01.189 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:00 smithi046 ceph-mon[29201]: pgmap v300: 97 pgs: 97 active+clean; 628 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 7.2 KiB/s wr, 17 op/s 2024-04-26T21:33:01.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:00 smithi142 ceph-mon[32944]: pgmap v300: 97 pgs: 97 active+clean; 628 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 7.2 KiB/s wr, 17 op/s 2024-04-26T21:33:01.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:00 smithi050 ceph-mon[32753]: pgmap v300: 97 pgs: 97 active+clean; 628 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 7.2 KiB/s wr, 17 op/s 2024-04-26T21:33:01.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.427+0000 7f6ba3a08700 1 -- 172.21.15.46:0/3583678111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b9c1027a0 msgr2=0x7f6b9c110210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:01.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.427+0000 7f6ba3a08700 1 --2- 172.21.15.46:0/3583678111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b9c1027a0 0x7f6b9c110210 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400b0a0 tx=0x7f6b9400b3b0 comp rx=0 tx=0).stop 2024-04-26T21:33:01.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.427+0000 7f6ba3a08700 1 -- 172.21.15.46:0/3583678111 shutdown_connections 2024-04-26T21:33:01.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.427+0000 7f6ba3a08700 1 --2- 172.21.15.46:0/3583678111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b9c1027a0 0x7f6b9c110210 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:01.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.427+0000 7f6ba3a08700 1 --2- 172.21.15.46:0/3583678111 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6b9c108500 0x7f6b9c1088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:01.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.427+0000 7f6ba3a08700 1 --2- 172.21.15.46:0/3583678111 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b9c101de0 0x7f6b9c102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:01.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.427+0000 7f6ba3a08700 1 -- 172.21.15.46:0/3583678111 >> 172.21.15.46:0/3583678111 conn(0x7f6b9c077640 msgr2=0x7f6b9c077a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:01.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.427+0000 7f6ba3a08700 1 -- 172.21.15.46:0/3583678111 shutdown_connections 2024-04-26T21:33:01.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.427+0000 7f6ba3a08700 1 -- 172.21.15.46:0/3583678111 wait complete. 2024-04-26T21:33:01.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.428+0000 7f6ba3a08700 1 Processor -- start 2024-04-26T21:33:01.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.428+0000 7f6ba3a08700 1 -- start start 2024-04-26T21:33:01.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.428+0000 7f6ba3a08700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b9c101de0 0x7f6b9c1a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:01.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.428+0000 7f6ba3a08700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6b9c1027a0 0x7f6b9c1a1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:01.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.428+0000 7f6ba3a08700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b9c108500 0x7f6b9c19bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:01.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.428+0000 7f6ba3a08700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9c113610 con 0x7f6b9c108500 2024-04-26T21:33:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.428+0000 7f6ba3a08700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9c113490 con 0x7f6b9c101de0 2024-04-26T21:33:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.428+0000 7f6ba3a08700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9c113790 con 0x7f6b9c1027a0 2024-04-26T21:33:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.428+0000 7f6ba1fa5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b9c108500 0x7f6b9c19bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.428+0000 7f6ba1fa5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b9c108500 0x7f6b9c19bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59790/0 (socket says 172.21.15.46:59790) 2024-04-26T21:33:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.428+0000 7f6ba1fa5700 1 -- 172.21.15.46:0/3390284571 learned_addr learned my addr 172.21.15.46:0/3390284571 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:33:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.428+0000 7f6ba17a4700 1 --2- 172.21.15.46:0/3390284571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b9c101de0 0x7f6b9c1a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.428+0000 7f6ba0fa3700 1 --2- 172.21.15.46:0/3390284571 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6b9c1027a0 0x7f6b9c1a1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.428+0000 7f6ba1fa5700 1 -- 172.21.15.46:0/3390284571 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6b9c1027a0 msgr2=0x7f6b9c1a1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.428+0000 7f6ba1fa5700 1 --2- 172.21.15.46:0/3390284571 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6b9c1027a0 0x7f6b9c1a1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.428+0000 7f6ba1fa5700 1 -- 172.21.15.46:0/3390284571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b9c101de0 msgr2=0x7f6b9c1a18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.428+0000 7f6ba1fa5700 1 --2- 172.21.15.46:0/3390284571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b9c101de0 0x7f6b9c1a18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.428+0000 7f6ba1fa5700 1 -- 172.21.15.46:0/3390284571 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b9c19c230 con 0x7f6b9c108500 2024-04-26T21:33:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.429+0000 7f6ba1fa5700 1 --2- 172.21.15.46:0/3390284571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b9c108500 0x7f6b9c19bad0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400b830 tx=0x7f6b94004d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.429+0000 7f6b927fc700 1 -- 172.21.15.46:0/3390284571 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b94021070 con 0x7f6b9c108500 2024-04-26T21:33:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.429+0000 7f6ba3a08700 1 -- 172.21.15.46:0/3390284571 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b9c19c4c0 con 0x7f6b9c108500 2024-04-26T21:33:01.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.429+0000 7f6b927fc700 1 -- 172.21.15.46:0/3390284571 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b94003c00 con 0x7f6b9c108500 2024-04-26T21:33:01.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.429+0000 7f6b927fc700 1 -- 172.21.15.46:0/3390284571 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b94004730 con 0x7f6b9c108500 2024-04-26T21:33:01.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.429+0000 7f6ba3a08700 1 -- 172.21.15.46:0/3390284571 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b9c1a8790 con 0x7f6b9c108500 2024-04-26T21:33:01.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.430+0000 7f6ba3a08700 1 -- 172.21.15.46:0/3390284571 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b9c06a110 con 0x7f6b9c108500 2024-04-26T21:33:01.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.431+0000 7f6b927fc700 1 -- 172.21.15.46:0/3390284571 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6b94007a00 con 0x7f6b9c108500 2024-04-26T21:33:01.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.431+0000 7f6b927fc700 1 --2- 172.21.15.46:0/3390284571 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6b880a3cd0 0x7f6b880a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:01.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.432+0000 7f6ba17a4700 1 --2- 172.21.15.46:0/3390284571 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6b880a3cd0 0x7f6b880a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:01.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.432+0000 7f6b927fc700 1 -- 172.21.15.46:0/3390284571 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f6b9400f020 con 0x7f6b9c108500 2024-04-26T21:33:01.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.432+0000 7f6ba17a4700 1 --2- 172.21.15.46:0/3390284571 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6b880a3cd0 0x7f6b880a6190 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6b98005950 tx=0x7f6b980058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:01.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.436+0000 7f6b927fc700 1 -- 172.21.15.46:0/3390284571 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6b9402a020 con 0x7f6b9c108500 2024-04-26T21:33:01.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.645+0000 7f6ba3a08700 1 -- 172.21.15.46:0/3390284571 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6b9c1a8960 con 0x7f6b9c108500 2024-04-26T21:33:01.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.646+0000 7f6b927fc700 1 -- 172.21.15.46:0/3390284571 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22197 (secure 0 0 0) 0x7f6b9409e590 con 0x7f6b9c108500 2024-04-26T21:33:01.647 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:33:01.648 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":85,"fsid":"36c7013a-0413-11ef-bc93-c7b262605968","created":"2024-04-26T21:24:30.481156+0000","modified":"2024-04-26T21:32:45.847101+0000","last_up_change":"2024-04-26T21:31:50.100491+0000","last_in_change":"2024-04-26T21:31:29.643428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:27:57.935988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:32:42.195211+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:32:43.162114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f53c8702-e4da-435f-a9dc-3911d0d36ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6803","nonce":2762665125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6805","nonce":2762665125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6809","nonce":2762665125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6807","nonce":2762665125}]},"public_addr":"172.21.15.46:6803/2762665125","cluster_addr":"172.21.15.46:6805/2762665125","heartbeat_back_addr":"172.21.15.46:6809/2762665125","heartbeat_front_addr":"172.21.15.46:6807/2762665125","state":["exists","up"]},{"osd":1,"uuid":"fc4d6f52-fc8a-4efd-80c9-c092b57e7f42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6801","nonce":1628008970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6803","nonce":1628008970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6807","nonce":1628008970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6805","nonce":1628008970}]},"public_addr":"172.21.15.50:6801/1628008970","cluster_addr":"172.21.15.50:6803/1628008970","heartbeat_back_addr":"172.21.15.50:6807/1628008970","heartbeat_front_addr":"172.21.15.50:6805/1628008970","state":["exists","up"]},{"osd":2,"uuid":"26d1baa1-4ea1-4b7e-9fcf-4f8bbf7329c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6801","nonce":1513279137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6803","nonce":1513279137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6807","nonce":1513279137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6805","nonce":1513279137}]},"public_addr":"172.21.15.142:6801/1513279137","cluster_addr":"172.21.15.142:6803/1513279137","heartbeat_back_addr":"172.21.15.142:6807/1513279137","heartbeat_front_addr":"172.21.15.142:6805/1513279137","state":["exists","up"]},{"osd":3,"uuid":"34d2401f-5fcc-491c-8fcf-c3c492611f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6811","nonce":2435181760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6813","nonce":2435181760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6817","nonce":2435181760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6815","nonce":2435181760}]},"public_addr":"172.21.15.46:6811/2435181760","cluster_addr":"172.21.15.46:6813/2435181760","heartbeat_back_addr":"172.21.15.46:6817/2435181760","heartbeat_front_addr":"172.21.15.46:6815/2435181760","state":["exists","up"]},{"osd":4,"uuid":"986bb709-e4cf-400d-982e-c3f5ac413e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6809","nonce":3177333250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6811","nonce":3177333250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6815","nonce":3177333250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6813","nonce":3177333250}]},"public_addr":"172.21.15.50:6809/3177333250","cluster_addr":"172.21.15.50:6811/3177333250","heartbeat_back_addr":"172.21.15.50:6815/3177333250","heartbeat_front_addr":"172.21.15.50:6813/3177333250","state":["exists","up"]},{"osd":5,"uuid":"b7d65851-ebe1-45e2-833d-8022446f2698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6809","nonce":4113107737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6811","nonce":4113107737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6815","nonce":4113107737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6813","nonce":4113107737}]},"public_addr":"172.21.15.142:6809/4113107737","cluster_addr":"172.21.15.142:6811/4113107737","heartbeat_back_addr":"172.21.15.142:6815/4113107737","heartbeat_front_addr":"172.21.15.142:6813/4113107737","state":["exists","up"]},{"osd":6,"uuid":"2a3e96d7-d5bf-44ff-9134-29f1420c0e4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6819","nonce":2325162882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6821","nonce":2325162882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6825","nonce":2325162882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6823","nonce":2325162882}]},"public_addr":"172.21.15.46:6819/2325162882","cluster_addr":"172.21.15.46:6821/2325162882","heartbeat_back_addr":"172.21.15.46:6825/2325162882","heartbeat_front_addr":"172.21.15.46:6823/2325162882","state":["exists","up"]},{"osd":7,"uuid":"e53ae885-23b6-46fd-b4b1-c62f31a6921f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6817","nonce":820310228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6819","nonce":820310228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6823","nonce":820310228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6821","nonce":820310228}]},"public_addr":"172.21.15.50:6817/820310228","cluster_addr":"172.21.15.50:6819/820310228","heartbeat_back_addr":"172.21.15.50:6823/820310228","heartbeat_front_addr":"172.21.15.50:6821/820310228","state":["exists","up"]},{"osd":8,"uuid":"356e2692-16a9-4045-b58f-c8f9cb5d7241","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6817","nonce":1824918711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6819","nonce":1824918711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6823","nonce":1824918711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6821","nonce":1824918711}]},"public_addr":"172.21.15.142:6817/1824918711","cluster_addr":"172.21.15.142:6819/1824918711","heartbeat_back_addr":"172.21.15.142:6823/1824918711","heartbeat_front_addr":"172.21.15.142:6821/1824918711","state":["exists","up"]},{"osd":9,"uuid":"8e55570e-77e5-4892-8b9f-77dcd03a4eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6827","nonce":3601947736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6829","nonce":3601947736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6833","nonce":3601947736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6831","nonce":3601947736}]},"public_addr":"172.21.15.46:6827/3601947736","cluster_addr":"172.21.15.46:6829/3601947736","heartbeat_back_addr":"172.21.15.46:6833/3601947736","heartbeat_front_addr":"172.21.15.46:6831/3601947736","state":["exists","up"]},{"osd":10,"uuid":"366389fa-fb9a-408b-b5c2-343c054c5a64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6825","nonce":3135765006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6827","nonce":3135765006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6831","nonce":3135765006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6829","nonce":3135765006}]},"public_addr":"172.21.15.50:6825/3135765006","cluster_addr":"172.21.15.50:6827/3135765006","heartbeat_back_addr":"172.21.15.50:6831/3135765006","heartbeat_front_addr":"172.21.15.50:6829/3135765006","state":["exists","up"]},{"osd":11,"uuid":"7b6e3d93-59e1-4853-b4a2-261b728c47a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6825","nonce":1482712612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6827","nonce":1482712612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6831","nonce":1482712612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6829","nonce":1482712612}]},"public_addr":"172.21.15.142:6825/1482712612","cluster_addr":"172.21.15.142:6827/1482712612","heartbeat_back_addr":"172.21.15.142:6831/1482712612","heartbeat_front_addr":"172.21.15.142:6829/1482712612","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:08.082015+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:32.634928+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:55.978864+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:21.796863+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:46.862518+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:11.440432+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:37.764777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:03.374958+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:28.714052+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:55.761167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:22.330951+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:47.839334+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/2465242088":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6801/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6800/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/1381997090":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/3507268217":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2570024356":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/2691061938":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2775954801":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6800/2896098005":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6801/2896098005":"2024-04-27T21:25:02.078730+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:33:01.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.648+0000 7f6ba3a08700 1 -- 172.21.15.46:0/3390284571 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6b880a3cd0 msgr2=0x7f6b880a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:01.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.648+0000 7f6ba3a08700 1 --2- 172.21.15.46:0/3390284571 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6b880a3cd0 0x7f6b880a6190 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6b98005950 tx=0x7f6b980058e0 comp rx=0 tx=0).stop 2024-04-26T21:33:01.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.648+0000 7f6ba3a08700 1 -- 172.21.15.46:0/3390284571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b9c108500 msgr2=0x7f6b9c19bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:01.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.648+0000 7f6ba3a08700 1 --2- 172.21.15.46:0/3390284571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b9c108500 0x7f6b9c19bad0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400b830 tx=0x7f6b94004d90 comp rx=0 tx=0).stop 2024-04-26T21:33:01.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.649+0000 7f6ba3a08700 1 -- 172.21.15.46:0/3390284571 shutdown_connections 2024-04-26T21:33:01.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.649+0000 7f6ba3a08700 1 --2- 172.21.15.46:0/3390284571 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6b880a3cd0 0x7f6b880a6190 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:01.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.649+0000 7f6ba3a08700 1 --2- 172.21.15.46:0/3390284571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b9c108500 0x7f6b9c19bad0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:01.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.649+0000 7f6ba3a08700 1 --2- 172.21.15.46:0/3390284571 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b9c101de0 0x7f6b9c1a18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:01.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.649+0000 7f6ba3a08700 1 --2- 172.21.15.46:0/3390284571 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6b9c1027a0 0x7f6b9c1a1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:01.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.649+0000 7f6ba3a08700 1 -- 172.21.15.46:0/3390284571 >> 172.21.15.46:0/3390284571 conn(0x7f6b9c077640 msgr2=0x7f6b9c1008b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:01.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.649+0000 7f6ba3a08700 1 -- 172.21.15.46:0/3390284571 shutdown_connections 2024-04-26T21:33:01.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:01.649+0000 7f6ba3a08700 1 -- 172.21.15.46:0/3390284571 wait complete. 2024-04-26T21:33:02.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:01 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3390284571' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:33:02.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:01 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3390284571' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:33:02.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:01 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3390284571' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:33:02.471 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-26T21:33:02.471 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T21:33:02.727 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:33:03.114 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:02 smithi046 ceph-mon[29201]: pgmap v301: 97 pgs: 97 active+clean; 628 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 5.6 KiB/s wr, 12 op/s 2024-04-26T21:33:03.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:02 smithi142 ceph-mon[32944]: pgmap v301: 97 pgs: 97 active+clean; 628 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 5.6 KiB/s wr, 12 op/s 2024-04-26T21:33:03.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:02 smithi050 ceph-mon[32753]: pgmap v301: 97 pgs: 97 active+clean; 628 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 5.6 KiB/s wr, 12 op/s 2024-04-26T21:33:04.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.405+0000 7fbe99a2b700 1 -- 172.21.15.46:0/1857473228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe9410fec0 msgr2=0x7fbe941122b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:04.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.405+0000 7fbe99a2b700 1 --2- 172.21.15.46:0/1857473228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe9410fec0 0x7fbe941122b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c00b0a0 tx=0x7fbe8c00b3b0 comp rx=0 tx=0).stop 2024-04-26T21:33:04.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.405+0000 7fbe99a2b700 1 -- 172.21.15.46:0/1857473228 shutdown_connections 2024-04-26T21:33:04.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.405+0000 7fbe99a2b700 1 --2- 172.21.15.46:0/1857473228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe9410fec0 0x7fbe941122b0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:04.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.405+0000 7fbe99a2b700 1 --2- 172.21.15.46:0/1857473228 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe9406b9c0 0x7fbe9406bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:04.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.405+0000 7fbe99a2b700 1 --2- 172.21.15.46:0/1857473228 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe9406c2e0 0x7fbe9410f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:04.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.405+0000 7fbe99a2b700 1 -- 172.21.15.46:0/1857473228 >> 172.21.15.46:0/1857473228 conn(0x7fbe94079290 msgr2=0x7fbe940796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:04.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.405+0000 7fbe99a2b700 1 -- 172.21.15.46:0/1857473228 shutdown_connections 2024-04-26T21:33:04.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.405+0000 7fbe99a2b700 1 -- 172.21.15.46:0/1857473228 wait complete. 2024-04-26T21:33:04.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.406+0000 7fbe99a2b700 1 Processor -- start 2024-04-26T21:33:04.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.406+0000 7fbe99a2b700 1 -- start start 2024-04-26T21:33:04.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.407+0000 7fbe99a2b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe9406b9c0 0x7fbe941a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:04.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.407+0000 7fbe99a2b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe9406c2e0 0x7fbe941a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:04.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.407+0000 7fbe99a2b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe9410fec0 0x7fbe9419bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:04.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.407+0000 7fbe99a2b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbe941157e0 con 0x7fbe9406b9c0 2024-04-26T21:33:04.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.407+0000 7fbe99a2b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbe94115660 con 0x7fbe9410fec0 2024-04-26T21:33:04.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.407+0000 7fbe99a2b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbe94115960 con 0x7fbe9406c2e0 2024-04-26T21:33:04.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.407+0000 7fbe92ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe9406b9c0 0x7fbe941a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:04.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.407+0000 7fbe92ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe9406b9c0 0x7fbe941a1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59804/0 (socket says 172.21.15.46:59804) 2024-04-26T21:33:04.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.407+0000 7fbe92ffd700 1 -- 172.21.15.46:0/1115088768 learned_addr learned my addr 172.21.15.46:0/1115088768 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:33:04.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.407+0000 7fbe937fe700 1 --2- 172.21.15.46:0/1115088768 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe9410fec0 0x7fbe9419bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:04.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.407+0000 7fbe927fc700 1 --2- 172.21.15.46:0/1115088768 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe9406c2e0 0x7fbe941a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:04.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.407+0000 7fbe92ffd700 1 -- 172.21.15.46:0/1115088768 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe9406c2e0 msgr2=0x7fbe941a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:04.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.407+0000 7fbe92ffd700 1 --2- 172.21.15.46:0/1115088768 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe9406c2e0 0x7fbe941a1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:04.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.407+0000 7fbe92ffd700 1 -- 172.21.15.46:0/1115088768 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe9410fec0 msgr2=0x7fbe9419bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:04.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.407+0000 7fbe92ffd700 1 --2- 172.21.15.46:0/1115088768 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe9410fec0 0x7fbe9419bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:04.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.407+0000 7fbe92ffd700 1 -- 172.21.15.46:0/1115088768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe9419c240 con 0x7fbe9406b9c0 2024-04-26T21:33:04.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.407+0000 7fbe937fe700 1 --2- 172.21.15.46:0/1115088768 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe9410fec0 0x7fbe9419bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:33:04.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.407+0000 7fbe92ffd700 1 --2- 172.21.15.46:0/1115088768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe9406b9c0 0x7fbe941a1920 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400e9c0 tx=0x7fbe8400ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:04.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.407+0000 7fbe98a29700 1 -- 172.21.15.46:0/1115088768 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8400cbd0 con 0x7fbe9406b9c0 2024-04-26T21:33:04.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.408+0000 7fbe98a29700 1 -- 172.21.15.46:0/1115088768 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe8400cd70 con 0x7fbe9406b9c0 2024-04-26T21:33:04.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.408+0000 7fbe99a2b700 1 -- 172.21.15.46:0/1115088768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe9419c530 con 0x7fbe9406b9c0 2024-04-26T21:33:04.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.408+0000 7fbe98a29700 1 -- 172.21.15.46:0/1115088768 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe84010430 con 0x7fbe9406b9c0 2024-04-26T21:33:04.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.408+0000 7fbe99a2b700 1 -- 172.21.15.46:0/1115088768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe941a87f0 con 0x7fbe9406b9c0 2024-04-26T21:33:04.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.409+0000 7fbe99a2b700 1 -- 172.21.15.46:0/1115088768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe9406a110 con 0x7fbe9406b9c0 2024-04-26T21:33:04.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.410+0000 7fbe98a29700 1 -- 172.21.15.46:0/1115088768 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbe84004750 con 0x7fbe9406b9c0 2024-04-26T21:33:04.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.411+0000 7fbe98a29700 1 --2- 172.21.15.46:0/1115088768 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fbe800a3cd0 0x7fbe800a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:04.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.411+0000 7fbe927fc700 1 --2- 172.21.15.46:0/1115088768 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fbe800a3cd0 0x7fbe800a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:04.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.411+0000 7fbe98a29700 1 -- 172.21.15.46:0/1115088768 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fbe840d08a0 con 0x7fbe9406b9c0 2024-04-26T21:33:04.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.412+0000 7fbe927fc700 1 --2- 172.21.15.46:0/1115088768 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fbe800a3cd0 0x7fbe800a6190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c005950 tx=0x7fbe7c0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:04.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.416+0000 7fbe98a29700 1 -- 172.21.15.46:0/1115088768 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbe8409d1d0 con 0x7fbe9406b9c0 2024-04-26T21:33:04.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.625+0000 7fbe99a2b700 1 -- 172.21.15.46:0/1115088768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbe9402e490 con 0x7fbe9406b9c0 2024-04-26T21:33:04.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.626+0000 7fbe98a29700 1 -- 172.21.15.46:0/1115088768 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22197 (secure 0 0 0) 0x7fbe8409cad0 con 0x7fbe9406b9c0 2024-04-26T21:33:04.628 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:33:04.628 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":85,"fsid":"36c7013a-0413-11ef-bc93-c7b262605968","created":"2024-04-26T21:24:30.481156+0000","modified":"2024-04-26T21:32:45.847101+0000","last_up_change":"2024-04-26T21:31:50.100491+0000","last_in_change":"2024-04-26T21:31:29.643428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:27:57.935988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:32:42.195211+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:32:43.162114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f53c8702-e4da-435f-a9dc-3911d0d36ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6803","nonce":2762665125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6805","nonce":2762665125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6809","nonce":2762665125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6807","nonce":2762665125}]},"public_addr":"172.21.15.46:6803/2762665125","cluster_addr":"172.21.15.46:6805/2762665125","heartbeat_back_addr":"172.21.15.46:6809/2762665125","heartbeat_front_addr":"172.21.15.46:6807/2762665125","state":["exists","up"]},{"osd":1,"uuid":"fc4d6f52-fc8a-4efd-80c9-c092b57e7f42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6801","nonce":1628008970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6803","nonce":1628008970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6807","nonce":1628008970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6805","nonce":1628008970}]},"public_addr":"172.21.15.50:6801/1628008970","cluster_addr":"172.21.15.50:6803/1628008970","heartbeat_back_addr":"172.21.15.50:6807/1628008970","heartbeat_front_addr":"172.21.15.50:6805/1628008970","state":["exists","up"]},{"osd":2,"uuid":"26d1baa1-4ea1-4b7e-9fcf-4f8bbf7329c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6801","nonce":1513279137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6803","nonce":1513279137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6807","nonce":1513279137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6805","nonce":1513279137}]},"public_addr":"172.21.15.142:6801/1513279137","cluster_addr":"172.21.15.142:6803/1513279137","heartbeat_back_addr":"172.21.15.142:6807/1513279137","heartbeat_front_addr":"172.21.15.142:6805/1513279137","state":["exists","up"]},{"osd":3,"uuid":"34d2401f-5fcc-491c-8fcf-c3c492611f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6811","nonce":2435181760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6813","nonce":2435181760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6817","nonce":2435181760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6815","nonce":2435181760}]},"public_addr":"172.21.15.46:6811/2435181760","cluster_addr":"172.21.15.46:6813/2435181760","heartbeat_back_addr":"172.21.15.46:6817/2435181760","heartbeat_front_addr":"172.21.15.46:6815/2435181760","state":["exists","up"]},{"osd":4,"uuid":"986bb709-e4cf-400d-982e-c3f5ac413e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6809","nonce":3177333250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6811","nonce":3177333250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6815","nonce":3177333250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6813","nonce":3177333250}]},"public_addr":"172.21.15.50:6809/3177333250","cluster_addr":"172.21.15.50:6811/3177333250","heartbeat_back_addr":"172.21.15.50:6815/3177333250","heartbeat_front_addr":"172.21.15.50:6813/3177333250","state":["exists","up"]},{"osd":5,"uuid":"b7d65851-ebe1-45e2-833d-8022446f2698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6809","nonce":4113107737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6811","nonce":4113107737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6815","nonce":4113107737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6813","nonce":4113107737}]},"public_addr":"172.21.15.142:6809/4113107737","cluster_addr":"172.21.15.142:6811/4113107737","heartbeat_back_addr":"172.21.15.142:6815/4113107737","heartbeat_front_addr":"172.21.15.142:6813/4113107737","state":["exists","up"]},{"osd":6,"uuid":"2a3e96d7-d5bf-44ff-9134-29f1420c0e4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6819","nonce":2325162882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6821","nonce":2325162882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6825","nonce":2325162882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6823","nonce":2325162882}]},"public_addr":"172.21.15.46:6819/2325162882","cluster_addr":"172.21.15.46:6821/2325162882","heartbeat_back_addr":"172.21.15.46:6825/2325162882","heartbeat_front_addr":"172.21.15.46:6823/2325162882","state":["exists","up"]},{"osd":7,"uuid":"e53ae885-23b6-46fd-b4b1-c62f31a6921f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6817","nonce":820310228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6819","nonce":820310228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6823","nonce":820310228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6821","nonce":820310228}]},"public_addr":"172.21.15.50:6817/820310228","cluster_addr":"172.21.15.50:6819/820310228","heartbeat_back_addr":"172.21.15.50:6823/820310228","heartbeat_front_addr":"172.21.15.50:6821/820310228","state":["exists","up"]},{"osd":8,"uuid":"356e2692-16a9-4045-b58f-c8f9cb5d7241","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6817","nonce":1824918711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6819","nonce":1824918711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6823","nonce":1824918711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6821","nonce":1824918711}]},"public_addr":"172.21.15.142:6817/1824918711","cluster_addr":"172.21.15.142:6819/1824918711","heartbeat_back_addr":"172.21.15.142:6823/1824918711","heartbeat_front_addr":"172.21.15.142:6821/1824918711","state":["exists","up"]},{"osd":9,"uuid":"8e55570e-77e5-4892-8b9f-77dcd03a4eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6827","nonce":3601947736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6829","nonce":3601947736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6833","nonce":3601947736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6831","nonce":3601947736}]},"public_addr":"172.21.15.46:6827/3601947736","cluster_addr":"172.21.15.46:6829/3601947736","heartbeat_back_addr":"172.21.15.46:6833/3601947736","heartbeat_front_addr":"172.21.15.46:6831/3601947736","state":["exists","up"]},{"osd":10,"uuid":"366389fa-fb9a-408b-b5c2-343c054c5a64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6825","nonce":3135765006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6827","nonce":3135765006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6831","nonce":3135765006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6829","nonce":3135765006}]},"public_addr":"172.21.15.50:6825/3135765006","cluster_addr":"172.21.15.50:6827/3135765006","heartbeat_back_addr":"172.21.15.50:6831/3135765006","heartbeat_front_addr":"172.21.15.50:6829/3135765006","state":["exists","up"]},{"osd":11,"uuid":"7b6e3d93-59e1-4853-b4a2-261b728c47a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6825","nonce":1482712612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6827","nonce":1482712612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6831","nonce":1482712612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6829","nonce":1482712612}]},"public_addr":"172.21.15.142:6825/1482712612","cluster_addr":"172.21.15.142:6827/1482712612","heartbeat_back_addr":"172.21.15.142:6831/1482712612","heartbeat_front_addr":"172.21.15.142:6829/1482712612","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:08.082015+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:32.634928+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:55.978864+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:21.796863+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:46.862518+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:11.440432+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:37.764777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:03.374958+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:28.714052+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:55.761167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:22.330951+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:47.839334+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/2465242088":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6801/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6800/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/1381997090":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/3507268217":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2570024356":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/2691061938":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2775954801":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6800/2896098005":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6801/2896098005":"2024-04-27T21:25:02.078730+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:33:04.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.628+0000 7fbe99a2b700 1 -- 172.21.15.46:0/1115088768 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fbe800a3cd0 msgr2=0x7fbe800a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:04.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.628+0000 7fbe99a2b700 1 --2- 172.21.15.46:0/1115088768 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fbe800a3cd0 0x7fbe800a6190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c005950 tx=0x7fbe7c0058e0 comp rx=0 tx=0).stop 2024-04-26T21:33:04.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.628+0000 7fbe99a2b700 1 -- 172.21.15.46:0/1115088768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe9406b9c0 msgr2=0x7fbe941a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:04.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.628+0000 7fbe99a2b700 1 --2- 172.21.15.46:0/1115088768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe9406b9c0 0x7fbe941a1920 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400e9c0 tx=0x7fbe8400ed80 comp rx=0 tx=0).stop 2024-04-26T21:33:04.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.628+0000 7fbe99a2b700 1 -- 172.21.15.46:0/1115088768 shutdown_connections 2024-04-26T21:33:04.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.628+0000 7fbe99a2b700 1 --2- 172.21.15.46:0/1115088768 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fbe800a3cd0 0x7fbe800a6190 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:04.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.628+0000 7fbe99a2b700 1 --2- 172.21.15.46:0/1115088768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe9406b9c0 0x7fbe941a1920 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:04.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.628+0000 7fbe99a2b700 1 --2- 172.21.15.46:0/1115088768 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbe9410fec0 0x7fbe9419bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:04.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.628+0000 7fbe99a2b700 1 --2- 172.21.15.46:0/1115088768 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbe9406c2e0 0x7fbe941a1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:04.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.628+0000 7fbe99a2b700 1 -- 172.21.15.46:0/1115088768 >> 172.21.15.46:0/1115088768 conn(0x7fbe94079290 msgr2=0x7fbe94111330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:04.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.629+0000 7fbe99a2b700 1 -- 172.21.15.46:0/1115088768 shutdown_connections 2024-04-26T21:33:04.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:04.629+0000 7fbe99a2b700 1 -- 172.21.15.46:0/1115088768 wait complete. 2024-04-26T21:33:05.262 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:05 smithi046 ceph-mon[29201]: pgmap v302: 97 pgs: 97 active+clean; 628 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 4.8 KiB/s wr, 10 op/s 2024-04-26T21:33:05.262 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:05 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1115088768' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:33:05.291 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-26T21:33:05.291 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-26T21:33:05.292 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-26T21:33:05.292 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-26T21:33:05.292 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-26T21:33:05.293 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-26T21:33:05.293 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-26T21:33:05.293 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-26T21:33:05.294 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-26T21:33:05.294 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-26T21:33:05.295 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-26T21:33:05.295 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-26T21:33:05.298 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:05 smithi142 ceph-mon[32944]: pgmap v302: 97 pgs: 97 active+clean; 628 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 4.8 KiB/s wr, 10 op/s 2024-04-26T21:33:05.298 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:05 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/1115088768' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:33:05.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:05 smithi050 ceph-mon[32753]: pgmap v302: 97 pgs: 97 active+clean; 628 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 4.8 KiB/s wr, 10 op/s 2024-04-26T21:33:05.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:05 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/1115088768' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:33:06.111 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:33:07.293 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:07 smithi142 ceph-mon[32944]: pgmap v303: 97 pgs: 97 active+clean; 628 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 4.6 KiB/s wr, 11 op/s 2024-04-26T21:33:07.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:07 smithi046 ceph-mon[29201]: pgmap v303: 97 pgs: 97 active+clean; 628 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 4.6 KiB/s wr, 11 op/s 2024-04-26T21:33:07.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:07 smithi050 ceph-mon[32753]: pgmap v303: 97 pgs: 97 active+clean; 628 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 4.6 KiB/s wr, 11 op/s 2024-04-26T21:33:07.380 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:33:07.380 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:33:07.387 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:33:07.387 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:33:07.393 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:33:07.396 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:33:07.396 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:33:07.400 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:33:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:33:07.413 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:33:07.461 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:33:09.248 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:09 smithi046 ceph-mon[29201]: pgmap v304: 97 pgs: 97 active+clean; 628 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-26T21:33:09.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:09 smithi142 ceph-mon[32944]: pgmap v304: 97 pgs: 97 active+clean; 628 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-26T21:33:09.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:09 smithi050 ceph-mon[32753]: pgmap v304: 97 pgs: 97 active+clean; 628 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-26T21:33:11.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:11 smithi142 ceph-mon[32944]: pgmap v305: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 8 op/s 2024-04-26T21:33:11.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:11 smithi046 ceph-mon[29201]: pgmap v305: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 8 op/s 2024-04-26T21:33:11.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:11 smithi050 ceph-mon[32753]: pgmap v305: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 8 op/s 2024-04-26T21:33:13.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:13 smithi046 ceph-mon[29201]: pgmap v306: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T21:33:13.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:13 smithi142 ceph-mon[32944]: pgmap v306: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T21:33:13.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:13 smithi050 ceph-mon[32753]: pgmap v306: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T21:33:15.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:15 smithi142 ceph-mon[32944]: pgmap v307: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T21:33:15.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:15 smithi046 ceph-mon[29201]: pgmap v307: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T21:33:15.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:15 smithi050 ceph-mon[32753]: pgmap v307: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T21:33:16.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.033+0000 7fbfa5fc3700 1 -- 172.21.15.46:0/733668964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfa01094a0 msgr2=0x7fbfa0109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:16.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.033+0000 7fbfa5fc3700 1 --2- 172.21.15.46:0/733668964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfa01094a0 0x7fbfa0109880 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbf90009990 tx=0x7fbf90009ca0 comp rx=0 tx=0).stop 2024-04-26T21:33:16.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.033+0000 7fbfa5fc3700 1 -- 172.21.15.46:0/733668964 shutdown_connections 2024-04-26T21:33:16.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.033+0000 7fbfa5fc3700 1 --2- 172.21.15.46:0/733668964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfa01094a0 0x7fbfa0109880 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:16.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.033+0000 7fbfa5fc3700 1 --2- 172.21.15.46:0/733668964 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfa006c510 0x7fbfa01102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:16.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.033+0000 7fbfa5fc3700 1 --2- 172.21.15.46:0/733668964 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbfa006bb70 0x7fbfa006bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:16.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.033+0000 7fbfa5fc3700 1 -- 172.21.15.46:0/733668964 >> 172.21.15.46:0/733668964 conn(0x7fbfa00792f0 msgr2=0x7fbfa0079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:16.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.033+0000 7fbfa5fc3700 1 -- 172.21.15.46:0/733668964 shutdown_connections 2024-04-26T21:33:16.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.033+0000 7fbfa5fc3700 1 -- 172.21.15.46:0/733668964 wait complete. 2024-04-26T21:33:16.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.033+0000 7fbfa5fc3700 1 Processor -- start 2024-04-26T21:33:16.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.033+0000 7fbfa5fc3700 1 -- start start 2024-04-26T21:33:16.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.034+0000 7fbfa5fc3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfa006bb70 0x7fbfa019f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:16.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.034+0000 7fbfa5fc3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfa006c510 0x7fbfa019fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:16.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.034+0000 7fbfa5fc3700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbfa01094a0 0x7fbfa01999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:16.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.034+0000 7fbfa5fc3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa0113610 con 0x7fbfa006c510 2024-04-26T21:33:16.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.034+0000 7fbfa5fc3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa0113490 con 0x7fbfa006bb70 2024-04-26T21:33:16.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.034+0000 7fbfa5fc3700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa0113790 con 0x7fbfa01094a0 2024-04-26T21:33:16.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.034+0000 7fbf97fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfa006c510 0x7fbfa019fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:16.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.034+0000 7fbf97fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfa006c510 0x7fbfa019fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44714/0 (socket says 172.21.15.46:44714) 2024-04-26T21:33:16.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.034+0000 7fbf97fff700 1 -- 172.21.15.46:0/3041922600 learned_addr learned my addr 172.21.15.46:0/3041922600 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:33:16.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.034+0000 7fbf9f7fe700 1 --2- 172.21.15.46:0/3041922600 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfa006bb70 0x7fbfa019f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:16.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.034+0000 7fbf9ffff700 1 --2- 172.21.15.46:0/3041922600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbfa01094a0 0x7fbfa01999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:16.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.034+0000 7fbf97fff700 1 -- 172.21.15.46:0/3041922600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbfa01094a0 msgr2=0x7fbfa01999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:16.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.034+0000 7fbf97fff700 1 --2- 172.21.15.46:0/3041922600 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fbfa01094a0 0x7fbfa01999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:16.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.034+0000 7fbf97fff700 1 -- 172.21.15.46:0/3041922600 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfa006bb70 msgr2=0x7fbfa019f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:16.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.034+0000 7fbf97fff700 1 --2- 172.21.15.46:0/3041922600 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbfa006bb70 0x7fbfa019f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:16.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.034+0000 7fbf97fff700 1 -- 172.21.15.46:0/3041922600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa019a130 con 0x7fbfa006c510 2024-04-26T21:33:16.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.034+0000 7fbf97fff700 1 --2- 172.21.15.46:0/3041922600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfa006c510 0x7fbfa019fd50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbf8800b530 tx=0x7fbf8800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:16.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.034+0000 7fbf9d7fa700 1 -- 172.21.15.46:0/3041922600 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf88015020 con 0x7fbfa006c510 2024-04-26T21:33:16.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.035+0000 7fbfa5fc3700 1 -- 172.21.15.46:0/3041922600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfa019a420 con 0x7fbfa006c510 2024-04-26T21:33:16.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.035+0000 7fbf9d7fa700 1 -- 172.21.15.46:0/3041922600 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf88011ce0 con 0x7fbfa006c510 2024-04-26T21:33:16.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.035+0000 7fbf9d7fa700 1 -- 172.21.15.46:0/3041922600 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf88010d20 con 0x7fbfa006c510 2024-04-26T21:33:16.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.035+0000 7fbfa5fc3700 1 -- 172.21.15.46:0/3041922600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfa01a66e0 con 0x7fbfa006c510 2024-04-26T21:33:16.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.035+0000 7fbfa5fc3700 1 -- 172.21.15.46:0/3041922600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbfa0063bb0 con 0x7fbfa006c510 2024-04-26T21:33:16.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.037+0000 7fbf9d7fa700 1 -- 172.21.15.46:0/3041922600 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbf88018020 con 0x7fbfa006c510 2024-04-26T21:33:16.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.037+0000 7fbf9d7fa700 1 --2- 172.21.15.46:0/3041922600 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fbf8c0a3cd0 0x7fbf8c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:16.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.037+0000 7fbf9f7fe700 1 --2- 172.21.15.46:0/3041922600 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fbf8c0a3cd0 0x7fbf8c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:16.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.038+0000 7fbf9d7fa700 1 -- 172.21.15.46:0/3041922600 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fbf880d0230 con 0x7fbfa006c510 2024-04-26T21:33:16.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.038+0000 7fbf9d7fa700 1 --2- 172.21.15.46:0/3041922600 >> [v2:172.21.15.46:6802/2762665125,v1:172.21.15.46:6803/2762665125] conn(0x7fbf8c0ad4b0 0x7fbf8c0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:16.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.038+0000 7fbf9d7fa700 1 -- 172.21.15.46:0/3041922600 --> [v2:172.21.15.46:6802/2762665125,v1:172.21.15.46:6803/2762665125] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbf8c0aff80 con 0x7fbf8c0ad4b0 2024-04-26T21:33:16.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.038+0000 7fbf9d7fa700 1 -- 172.21.15.46:0/3041922600 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fbf880d0710 con 0x7fbfa006c510 2024-04-26T21:33:16.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.038+0000 7fbf9ffff700 1 --2- 172.21.15.46:0/3041922600 >> [v2:172.21.15.46:6802/2762665125,v1:172.21.15.46:6803/2762665125] conn(0x7fbf8c0ad4b0 0x7fbf8c0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:16.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.038+0000 7fbf9f7fe700 1 --2- 172.21.15.46:0/3041922600 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fbf8c0a3cd0 0x7fbf8c0a6190 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbf90006010 tx=0x7fbf9000b580 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:16.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.038+0000 7fbf9ffff700 1 --2- 172.21.15.46:0/3041922600 >> [v2:172.21.15.46:6802/2762665125,v1:172.21.15.46:6803/2762665125] conn(0x7fbf8c0ad4b0 0x7fbf8c0af8d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:16.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.039+0000 7fbf9d7fa700 1 -- 172.21.15.46:0/3041922600 <== osd.0 v2:172.21.15.46:6802/2762665125 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbf8c0aff80 con 0x7fbf8c0ad4b0 2024-04-26T21:33:16.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.055+0000 7fbfa5fc3700 1 -- 172.21.15.46:0/3041922600 --> [v2:172.21.15.46:6802/2762665125,v1:172.21.15.46:6803/2762665125] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbfa0063fb0 con 0x7fbf8c0ad4b0 2024-04-26T21:33:16.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.056+0000 7fbf9d7fa700 1 -- 172.21.15.46:0/3041922600 <== osd.0 v2:172.21.15.46:6802/2762665125 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fbfa0063fb0 con 0x7fbf8c0ad4b0 2024-04-26T21:33:16.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.056+0000 7fbfa5fc3700 1 -- 172.21.15.46:0/3041922600 >> [v2:172.21.15.46:6802/2762665125,v1:172.21.15.46:6803/2762665125] conn(0x7fbf8c0ad4b0 msgr2=0x7fbf8c0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:16.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.056+0000 7fbfa5fc3700 1 --2- 172.21.15.46:0/3041922600 >> [v2:172.21.15.46:6802/2762665125,v1:172.21.15.46:6803/2762665125] conn(0x7fbf8c0ad4b0 0x7fbf8c0af8d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:16.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.056+0000 7fbfa5fc3700 1 -- 172.21.15.46:0/3041922600 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fbf8c0a3cd0 msgr2=0x7fbf8c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:16.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.056+0000 7fbfa5fc3700 1 --2- 172.21.15.46:0/3041922600 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fbf8c0a3cd0 0x7fbf8c0a6190 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbf90006010 tx=0x7fbf9000b580 comp rx=0 tx=0).stop 2024-04-26T21:33:16.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.056+0000 7fbfa5fc3700 1 -- 172.21.15.46:0/3041922600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfa006c510 msgr2=0x7fbfa019fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:16.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.056+0000 7fbfa5fc3700 1 --2- 172.21.15.46:0/3041922600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfa006c510 0x7fbfa019fd50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbf8800b530 tx=0x7fbf8800b8f0 comp rx=0 tx=0).stop 2024-04-26T21:33:16.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.056+0000 7fbf9ffff700 1 -- 172.21.15.46:0/3041922600 reap_dead start 2024-04-26T21:33:16.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.056+0000 7fbfa5fc3700 1 -- 172.21.15.46:0/3041922600 shutdown_connections 2024-04-26T21:33:16.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.056+0000 7fbfa5fc3700 1 -- 172.21.15.46:0/3041922600 >> 172.21.15.46:0/3041922600 conn(0x7fbfa00792f0 msgr2=0x7fbfa0100500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:16.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.056+0000 7fbfa5fc3700 1 -- 172.21.15.46:0/3041922600 shutdown_connections 2024-04-26T21:33:16.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:16.056+0000 7fbfa5fc3700 1 -- 172.21.15.46:0/3041922600 wait complete. 2024-04-26T21:33:17.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:17 smithi142 ceph-mon[32944]: pgmap v308: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-26T21:33:17.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:17 smithi046 ceph-mon[29201]: pgmap v308: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-26T21:33:17.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:17 smithi050 ceph-mon[32753]: pgmap v308: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-26T21:33:17.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.610+0000 7f7dce8e6700 1 -- 172.21.15.46:0/3833002167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc8102ce0 msgr2=0x7f7dc810f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:17.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.610+0000 7f7dce8e6700 1 --2- 172.21.15.46:0/3833002167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc8102ce0 0x7f7dc810f1d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7dc000b0a0 tx=0x7f7dc000b3b0 comp rx=0 tx=0).stop 2024-04-26T21:33:17.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.610+0000 7f7dce8e6700 1 -- 172.21.15.46:0/3833002167 shutdown_connections 2024-04-26T21:33:17.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.610+0000 7f7dce8e6700 1 --2- 172.21.15.46:0/3833002167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc8102ce0 0x7f7dc810f1d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:17.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.610+0000 7f7dce8e6700 1 --2- 172.21.15.46:0/3833002167 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dc8108390 0x7f7dc8108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:17.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.610+0000 7f7dce8e6700 1 --2- 172.21.15.46:0/3833002167 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7dc8102320 0x7f7dc81027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:17.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.610+0000 7f7dce8e6700 1 -- 172.21.15.46:0/3833002167 >> 172.21.15.46:0/3833002167 conn(0x7f7dc80792d0 msgr2=0x7f7dc80796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:17.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.610+0000 7f7dce8e6700 1 -- 172.21.15.46:0/3833002167 shutdown_connections 2024-04-26T21:33:17.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.610+0000 7f7dce8e6700 1 -- 172.21.15.46:0/3833002167 wait complete. 2024-04-26T21:33:17.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.610+0000 7f7dce8e6700 1 Processor -- start 2024-04-26T21:33:17.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.611+0000 7f7dce8e6700 1 -- start start 2024-04-26T21:33:17.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.611+0000 7f7dce8e6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dc8102320 0x7f7dc8104000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:17.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.611+0000 7f7dce8e6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc8102ce0 0x7f7dc8104540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:17.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.611+0000 7f7dce8e6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7dc8108390 0x7f7dc8104a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:17.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.611+0000 7f7dce8e6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc8112530 con 0x7f7dc8102ce0 2024-04-26T21:33:17.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.611+0000 7f7dce8e6700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc81123b0 con 0x7f7dc8102320 2024-04-26T21:33:17.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.611+0000 7f7dce8e6700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc81126b0 con 0x7f7dc8108390 2024-04-26T21:33:17.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.611+0000 7f7dc77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc8102ce0 0x7f7dc8104540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:17.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.611+0000 7f7dc77fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc8102ce0 0x7f7dc8104540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44732/0 (socket says 172.21.15.46:44732) 2024-04-26T21:33:17.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.611+0000 7f7dc77fe700 1 -- 172.21.15.46:0/174071385 learned_addr learned my addr 172.21.15.46:0/174071385 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:33:17.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.611+0000 7f7dc7fff700 1 --2- 172.21.15.46:0/174071385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dc8102320 0x7f7dc8104000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:17.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.611+0000 7f7dcce83700 1 --2- 172.21.15.46:0/174071385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7dc8108390 0x7f7dc8104a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:17.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.611+0000 7f7dc77fe700 1 -- 172.21.15.46:0/174071385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7dc8108390 msgr2=0x7f7dc8104a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:17.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.611+0000 7f7dc77fe700 1 --2- 172.21.15.46:0/174071385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7dc8108390 0x7f7dc8104a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:17.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.611+0000 7f7dc77fe700 1 -- 172.21.15.46:0/174071385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dc8102320 msgr2=0x7f7dc8104000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:17.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.611+0000 7f7dc77fe700 1 --2- 172.21.15.46:0/174071385 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7dc8102320 0x7f7dc8104000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:17.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.611+0000 7f7dc77fe700 1 -- 172.21.15.46:0/174071385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dc81a8080 con 0x7f7dc8102ce0 2024-04-26T21:33:17.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.612+0000 7f7dcce83700 1 --2- 172.21.15.46:0/174071385 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7dc8108390 0x7f7dc8104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:33:17.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.612+0000 7f7dc77fe700 1 --2- 172.21.15.46:0/174071385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc8102ce0 0x7f7dc8104540 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc00b530 tx=0x7f7dbc00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:17.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.612+0000 7f7dc57fa700 1 -- 172.21.15.46:0/174071385 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc015020 con 0x7f7dc8102ce0 2024-04-26T21:33:17.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.612+0000 7f7dce8e6700 1 -- 172.21.15.46:0/174071385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dc81a8370 con 0x7f7dc8102ce0 2024-04-26T21:33:17.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.612+0000 7f7dc57fa700 1 -- 172.21.15.46:0/174071385 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dbc011ce0 con 0x7f7dc8102ce0 2024-04-26T21:33:17.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.612+0000 7f7dc57fa700 1 -- 172.21.15.46:0/174071385 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc010e40 con 0x7f7dc8102ce0 2024-04-26T21:33:17.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.612+0000 7f7dce8e6700 1 -- 172.21.15.46:0/174071385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dc81a88b0 con 0x7f7dc8102ce0 2024-04-26T21:33:17.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.613+0000 7f7dce8e6700 1 -- 172.21.15.46:0/174071385 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7dc8106410 con 0x7f7dc8102ce0 2024-04-26T21:33:17.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.614+0000 7f7dc57fa700 1 -- 172.21.15.46:0/174071385 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7dbc018020 con 0x7f7dc8102ce0 2024-04-26T21:33:17.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.615+0000 7f7dc57fa700 1 --2- 172.21.15.46:0/174071385 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7db00a3cd0 0x7f7db00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:17.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.615+0000 7f7dc7fff700 1 --2- 172.21.15.46:0/174071385 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7db00a3cd0 0x7f7db00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:17.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.615+0000 7f7dc57fa700 1 -- 172.21.15.46:0/174071385 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f7dbc0d0290 con 0x7f7dc8102ce0 2024-04-26T21:33:17.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.615+0000 7f7dc57fa700 1 --2- 172.21.15.46:0/174071385 >> [v2:172.21.15.50:6808/3177333250,v1:172.21.15.50:6809/3177333250] conn(0x7f7db00ad4b0 0x7f7db00af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:17.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.615+0000 7f7dc57fa700 1 -- 172.21.15.46:0/174071385 --> [v2:172.21.15.50:6808/3177333250,v1:172.21.15.50:6809/3177333250] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7db00aff80 con 0x7f7db00ad4b0 2024-04-26T21:33:17.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.615+0000 7f7dc57fa700 1 -- 172.21.15.46:0/174071385 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f7dbc0d0770 con 0x7f7dc8102ce0 2024-04-26T21:33:17.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.615+0000 7f7dc7fff700 1 --2- 172.21.15.46:0/174071385 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7db00a3cd0 0x7f7db00a6190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7db8005950 tx=0x7f7db800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:17.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.615+0000 7f7dcce83700 1 --2- 172.21.15.46:0/174071385 >> [v2:172.21.15.50:6808/3177333250,v1:172.21.15.50:6809/3177333250] conn(0x7f7db00ad4b0 0x7f7db00af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:17.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.616+0000 7f7dcce83700 1 --2- 172.21.15.46:0/174071385 >> [v2:172.21.15.50:6808/3177333250,v1:172.21.15.50:6809/3177333250] conn(0x7f7db00ad4b0 0x7f7db00af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:17.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.616+0000 7f7dc57fa700 1 -- 172.21.15.46:0/174071385 <== osd.4 v2:172.21.15.50:6808/3177333250 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7db00aff80 con 0x7f7db00ad4b0 2024-04-26T21:33:17.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.633+0000 7f7dce8e6700 1 -- 172.21.15.46:0/174071385 --> [v2:172.21.15.50:6808/3177333250,v1:172.21.15.50:6809/3177333250] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7dc8103a40 con 0x7f7db00ad4b0 2024-04-26T21:33:17.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.634+0000 7f7dc57fa700 1 -- 172.21.15.46:0/174071385 <== osd.4 v2:172.21.15.50:6808/3177333250 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7dc8103a40 con 0x7f7db00ad4b0 2024-04-26T21:33:17.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.634+0000 7f7dce8e6700 1 -- 172.21.15.46:0/174071385 >> [v2:172.21.15.50:6808/3177333250,v1:172.21.15.50:6809/3177333250] conn(0x7f7db00ad4b0 msgr2=0x7f7db00af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:17.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.634+0000 7f7dce8e6700 1 --2- 172.21.15.46:0/174071385 >> [v2:172.21.15.50:6808/3177333250,v1:172.21.15.50:6809/3177333250] conn(0x7f7db00ad4b0 0x7f7db00af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:17.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.634+0000 7f7dce8e6700 1 -- 172.21.15.46:0/174071385 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7db00a3cd0 msgr2=0x7f7db00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:17.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.634+0000 7f7dce8e6700 1 --2- 172.21.15.46:0/174071385 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7db00a3cd0 0x7f7db00a6190 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7db8005950 tx=0x7f7db800b410 comp rx=0 tx=0).stop 2024-04-26T21:33:17.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.634+0000 7f7dce8e6700 1 -- 172.21.15.46:0/174071385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc8102ce0 msgr2=0x7f7dc8104540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:17.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.634+0000 7f7dce8e6700 1 --2- 172.21.15.46:0/174071385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7dc8102ce0 0x7f7dc8104540 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc00b530 tx=0x7f7dbc00b840 comp rx=0 tx=0).stop 2024-04-26T21:33:17.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.634+0000 7f7dcce83700 1 -- 172.21.15.46:0/174071385 reap_dead start 2024-04-26T21:33:17.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.634+0000 7f7dce8e6700 1 -- 172.21.15.46:0/174071385 shutdown_connections 2024-04-26T21:33:17.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.634+0000 7f7dce8e6700 1 -- 172.21.15.46:0/174071385 >> 172.21.15.46:0/174071385 conn(0x7f7dc80792d0 msgr2=0x7f7dc81015e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:17.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.635+0000 7f7dce8e6700 1 -- 172.21.15.46:0/174071385 shutdown_connections 2024-04-26T21:33:17.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:17.635+0000 7f7dce8e6700 1 -- 172.21.15.46:0/174071385 wait complete. 2024-04-26T21:33:18.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.769+0000 7f70f3eed700 1 -- 172.21.15.46:0/2314582121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70ec109560 msgr2=0x7f70ec109940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:18.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.769+0000 7f70f3eed700 1 --2- 172.21.15.46:0/2314582121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70ec109560 0x7f70ec109940 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f70dc0099c0 tx=0x7f70dc009cd0 comp rx=0 tx=0).stop 2024-04-26T21:33:18.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.769+0000 7f70f3eed700 1 -- 172.21.15.46:0/2314582121 shutdown_connections 2024-04-26T21:33:18.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.769+0000 7f70f3eed700 1 --2- 172.21.15.46:0/2314582121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70ec109560 0x7f70ec109940 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:18.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.769+0000 7f70f3eed700 1 --2- 172.21.15.46:0/2314582121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f70ec103e80 0x7f70ec110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:18.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.769+0000 7f70f3eed700 1 --2- 172.21.15.46:0/2314582121 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70ec1034c0 0x7f70ec103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:18.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.769+0000 7f70f3eed700 1 -- 172.21.15.46:0/2314582121 >> 172.21.15.46:0/2314582121 conn(0x7f70ec07a5d0 msgr2=0x7f70ec07a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:18.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.769+0000 7f70f3eed700 1 -- 172.21.15.46:0/2314582121 shutdown_connections 2024-04-26T21:33:18.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.769+0000 7f70f3eed700 1 -- 172.21.15.46:0/2314582121 wait complete. 2024-04-26T21:33:18.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.770+0000 7f70f3eed700 1 Processor -- start 2024-04-26T21:33:18.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.770+0000 7f70f3eed700 1 -- start start 2024-04-26T21:33:18.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.770+0000 7f70f3eed700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70ec1034c0 0x7f70ec19f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:18.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.770+0000 7f70f3eed700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70ec103e80 0x7f70ec19fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:18.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.770+0000 7f70f3eed700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f70ec109560 0x7f70ec199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:18.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.770+0000 7f70f3eed700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f70ec113790 con 0x7f70ec103e80 2024-04-26T21:33:18.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.770+0000 7f70f3eed700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f70ec113610 con 0x7f70ec1034c0 2024-04-26T21:33:18.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.770+0000 7f70f3eed700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f70ec113910 con 0x7f70ec109560 2024-04-26T21:33:18.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.770+0000 7f70f1488700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70ec103e80 0x7f70ec19fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:18.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.770+0000 7f70f1488700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70ec103e80 0x7f70ec19fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44744/0 (socket says 172.21.15.46:44744) 2024-04-26T21:33:18.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.770+0000 7f70f1488700 1 -- 172.21.15.46:0/1517398787 learned_addr learned my addr 172.21.15.46:0/1517398787 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:33:18.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.770+0000 7f70f1c89700 1 --2- 172.21.15.46:0/1517398787 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70ec1034c0 0x7f70ec19f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:18.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.770+0000 7f70f248a700 1 --2- 172.21.15.46:0/1517398787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f70ec109560 0x7f70ec199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:18.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.770+0000 7f70f1488700 1 -- 172.21.15.46:0/1517398787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f70ec109560 msgr2=0x7f70ec199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:18.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.770+0000 7f70f1488700 1 --2- 172.21.15.46:0/1517398787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f70ec109560 0x7f70ec199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:18.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.770+0000 7f70f1488700 1 -- 172.21.15.46:0/1517398787 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70ec1034c0 msgr2=0x7f70ec19f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:18.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.770+0000 7f70f1488700 1 --2- 172.21.15.46:0/1517398787 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f70ec1034c0 0x7f70ec19f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:18.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.770+0000 7f70f1488700 1 -- 172.21.15.46:0/1517398787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70ec19a1f0 con 0x7f70ec103e80 2024-04-26T21:33:18.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.770+0000 7f70f248a700 1 --2- 172.21.15.46:0/1517398787 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f70ec109560 0x7f70ec199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:33:18.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.771+0000 7f70f1488700 1 --2- 172.21.15.46:0/1517398787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70ec103e80 0x7f70ec19fe10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f70e800b530 tx=0x7f70e800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:18.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.771+0000 7f70e2ffd700 1 -- 172.21.15.46:0/1517398787 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e8015020 con 0x7f70ec103e80 2024-04-26T21:33:18.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.771+0000 7f70f3eed700 1 -- 172.21.15.46:0/1517398787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70ec19a4e0 con 0x7f70ec103e80 2024-04-26T21:33:18.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.771+0000 7f70e2ffd700 1 -- 172.21.15.46:0/1517398787 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70e8011ce0 con 0x7f70ec103e80 2024-04-26T21:33:18.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.771+0000 7f70e2ffd700 1 -- 172.21.15.46:0/1517398787 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e8010e40 con 0x7f70ec103e80 2024-04-26T21:33:18.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.771+0000 7f70f3eed700 1 -- 172.21.15.46:0/1517398787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70ec1a67a0 con 0x7f70ec103e80 2024-04-26T21:33:18.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.772+0000 7f70f3eed700 1 -- 172.21.15.46:0/1517398787 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f70ec063b30 con 0x7f70ec103e80 2024-04-26T21:33:18.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.773+0000 7f70e2ffd700 1 -- 172.21.15.46:0/1517398787 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f70e8018020 con 0x7f70ec103e80 2024-04-26T21:33:18.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.774+0000 7f70e2ffd700 1 --2- 172.21.15.46:0/1517398787 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f70d80a3cd0 0x7f70d80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:18.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.774+0000 7f70f1c89700 1 --2- 172.21.15.46:0/1517398787 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f70d80a3cd0 0x7f70d80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:18.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.774+0000 7f70e2ffd700 1 -- 172.21.15.46:0/1517398787 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f70e80d0200 con 0x7f70ec103e80 2024-04-26T21:33:18.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.774+0000 7f70e2ffd700 1 --2- 172.21.15.46:0/1517398787 >> [v2:172.21.15.142:6824/1482712612,v1:172.21.15.142:6825/1482712612] conn(0x7f70d80ad4b0 0x7f70d80af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:18.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.774+0000 7f70e2ffd700 1 -- 172.21.15.46:0/1517398787 --> [v2:172.21.15.142:6824/1482712612,v1:172.21.15.142:6825/1482712612] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f70d80aff80 con 0x7f70d80ad4b0 2024-04-26T21:33:18.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.774+0000 7f70e2ffd700 1 -- 172.21.15.46:0/1517398787 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f70e80d06e0 con 0x7f70ec103e80 2024-04-26T21:33:18.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.774+0000 7f70f1c89700 1 --2- 172.21.15.46:0/1517398787 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f70d80a3cd0 0x7f70d80a6190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f70ec19abd0 tx=0x7f70dc005ad0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:18.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.774+0000 7f70f248a700 1 --2- 172.21.15.46:0/1517398787 >> [v2:172.21.15.142:6824/1482712612,v1:172.21.15.142:6825/1482712612] conn(0x7f70d80ad4b0 0x7f70d80af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:18.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.775+0000 7f70f248a700 1 --2- 172.21.15.46:0/1517398787 >> [v2:172.21.15.142:6824/1482712612,v1:172.21.15.142:6825/1482712612] conn(0x7f70d80ad4b0 0x7f70d80af8d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:18.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.776+0000 7f70e2ffd700 1 -- 172.21.15.46:0/1517398787 <== osd.11 v2:172.21.15.142:6824/1482712612 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f70d80aff80 con 0x7f70d80ad4b0 2024-04-26T21:33:18.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.792+0000 7f70f3eed700 1 -- 172.21.15.46:0/1517398787 --> [v2:172.21.15.142:6824/1482712612,v1:172.21.15.142:6825/1482712612] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f70ec063f30 con 0x7f70d80ad4b0 2024-04-26T21:33:18.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.792+0000 7f70e2ffd700 1 -- 172.21.15.46:0/1517398787 <== osd.11 v2:172.21.15.142:6824/1482712612 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f70ec063f30 con 0x7f70d80ad4b0 2024-04-26T21:33:18.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.793+0000 7f70f3eed700 1 -- 172.21.15.46:0/1517398787 >> [v2:172.21.15.142:6824/1482712612,v1:172.21.15.142:6825/1482712612] conn(0x7f70d80ad4b0 msgr2=0x7f70d80af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:18.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.793+0000 7f70f3eed700 1 --2- 172.21.15.46:0/1517398787 >> [v2:172.21.15.142:6824/1482712612,v1:172.21.15.142:6825/1482712612] conn(0x7f70d80ad4b0 0x7f70d80af8d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:18.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.793+0000 7f70f3eed700 1 -- 172.21.15.46:0/1517398787 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f70d80a3cd0 msgr2=0x7f70d80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:18.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.793+0000 7f70f3eed700 1 --2- 172.21.15.46:0/1517398787 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f70d80a3cd0 0x7f70d80a6190 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f70ec19abd0 tx=0x7f70dc005ad0 comp rx=0 tx=0).stop 2024-04-26T21:33:18.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.793+0000 7f70f3eed700 1 -- 172.21.15.46:0/1517398787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70ec103e80 msgr2=0x7f70ec19fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:18.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.793+0000 7f70f3eed700 1 --2- 172.21.15.46:0/1517398787 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70ec103e80 0x7f70ec19fe10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f70e800b530 tx=0x7f70e800b840 comp rx=0 tx=0).stop 2024-04-26T21:33:18.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.793+0000 7f70f248a700 1 -- 172.21.15.46:0/1517398787 reap_dead start 2024-04-26T21:33:18.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.793+0000 7f70f3eed700 1 -- 172.21.15.46:0/1517398787 shutdown_connections 2024-04-26T21:33:18.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.793+0000 7f70f3eed700 1 -- 172.21.15.46:0/1517398787 >> 172.21.15.46:0/1517398787 conn(0x7f70ec07a5d0 msgr2=0x7f70ec102200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:18.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.793+0000 7f70f3eed700 1 -- 172.21.15.46:0/1517398787 shutdown_connections 2024-04-26T21:33:18.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:18.793+0000 7f70f3eed700 1 -- 172.21.15.46:0/1517398787 wait complete. 2024-04-26T21:33:19.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:19 smithi142 ceph-mon[32944]: pgmap v309: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T21:33:19.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:19 smithi046 ceph-mon[29201]: pgmap v309: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T21:33:19.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:19 smithi050 ceph-mon[32753]: pgmap v309: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T21:33:19.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.786+0000 7f4ad92b6700 1 -- 172.21.15.46:0/113498985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ad41034a0 msgr2=0x7f4ad4103920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:19.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.786+0000 7f4ad92b6700 1 --2- 172.21.15.46:0/113498985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ad41034a0 0x7f4ad4103920 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8009990 tx=0x7f4ac8009ca0 comp rx=0 tx=0).stop 2024-04-26T21:33:19.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.786+0000 7f4ad92b6700 1 -- 172.21.15.46:0/113498985 shutdown_connections 2024-04-26T21:33:19.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.786+0000 7f4ad92b6700 1 --2- 172.21.15.46:0/113498985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4ad4103e60 0x7f4ad41102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:19.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.786+0000 7f4ad92b6700 1 --2- 172.21.15.46:0/113498985 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ad41094e0 0x7f4ad41098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:19.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.786+0000 7f4ad92b6700 1 --2- 172.21.15.46:0/113498985 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ad41034a0 0x7f4ad4103920 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:19.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.786+0000 7f4ad92b6700 1 -- 172.21.15.46:0/113498985 >> 172.21.15.46:0/113498985 conn(0x7f4ad407a590 msgr2=0x7f4ad407a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.786+0000 7f4ad92b6700 1 -- 172.21.15.46:0/113498985 shutdown_connections 2024-04-26T21:33:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.786+0000 7f4ad92b6700 1 -- 172.21.15.46:0/113498985 wait complete. 2024-04-26T21:33:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.787+0000 7f4ad92b6700 1 Processor -- start 2024-04-26T21:33:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.787+0000 7f4ad92b6700 1 -- start start 2024-04-26T21:33:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.787+0000 7f4ad92b6700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ad41034a0 0x7f4ad419f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.787+0000 7f4ad92b6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ad4103e60 0x7f4ad419fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.787+0000 7f4ad92b6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4ad41094e0 0x7f4ad4199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.787+0000 7f4ad92b6700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad4113660 con 0x7f4ad4103e60 2024-04-26T21:33:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.787+0000 7f4ad92b6700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad41134e0 con 0x7f4ad41094e0 2024-04-26T21:33:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.787+0000 7f4ad92b6700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad41137e0 con 0x7f4ad41034a0 2024-04-26T21:33:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.787+0000 7f4ad27fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ad4103e60 0x7f4ad419fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.788+0000 7f4ad27fc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ad4103e60 0x7f4ad419fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44760/0 (socket says 172.21.15.46:44760) 2024-04-26T21:33:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.788+0000 7f4ad27fc700 1 -- 172.21.15.46:0/707137040 learned_addr learned my addr 172.21.15.46:0/707137040 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:33:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.788+0000 7f4ad2ffd700 1 --2- 172.21.15.46:0/707137040 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ad41034a0 0x7f4ad419f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:19.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.788+0000 7f4ad37fe700 1 --2- 172.21.15.46:0/707137040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4ad41094e0 0x7f4ad4199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.788+0000 7f4ad27fc700 1 -- 172.21.15.46:0/707137040 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ad41034a0 msgr2=0x7f4ad419f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.788+0000 7f4ad27fc700 1 --2- 172.21.15.46:0/707137040 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ad41034a0 0x7f4ad419f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.788+0000 7f4ad27fc700 1 -- 172.21.15.46:0/707137040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4ad41094e0 msgr2=0x7f4ad4199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.788+0000 7f4ad27fc700 1 --2- 172.21.15.46:0/707137040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4ad41094e0 0x7f4ad4199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.788+0000 7f4ad27fc700 1 -- 172.21.15.46:0/707137040 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ad419a0c0 con 0x7f4ad4103e60 2024-04-26T21:33:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.788+0000 7f4ad27fc700 1 --2- 172.21.15.46:0/707137040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ad4103e60 0x7f4ad419fce0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4ac80057f0 tx=0x7f4ac8003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.788+0000 7f4abbfff700 1 -- 172.21.15.46:0/707137040 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac8027070 con 0x7f4ad4103e60 2024-04-26T21:33:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.788+0000 7f4abbfff700 1 -- 172.21.15.46:0/707137040 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ac8004020 con 0x7f4ad4103e60 2024-04-26T21:33:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.788+0000 7f4ad92b6700 1 -- 172.21.15.46:0/707137040 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ad419a350 con 0x7f4ad4103e60 2024-04-26T21:33:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.788+0000 7f4abbfff700 1 -- 172.21.15.46:0/707137040 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac8022d60 con 0x7f4ad4103e60 2024-04-26T21:33:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.788+0000 7f4ad92b6700 1 -- 172.21.15.46:0/707137040 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ad41a6670 con 0x7f4ad4103e60 2024-04-26T21:33:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.789+0000 7f4ad92b6700 1 -- 172.21.15.46:0/707137040 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4ad4063b30 con 0x7f4ad4103e60 2024-04-26T21:33:19.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.790+0000 7f4abbfff700 1 -- 172.21.15.46:0/707137040 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4ac802a070 con 0x7f4ad4103e60 2024-04-26T21:33:19.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.791+0000 7f4abbfff700 1 --2- 172.21.15.46:0/707137040 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4abc0a3cd0 0x7f4abc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:19.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.791+0000 7f4ad2ffd700 1 --2- 172.21.15.46:0/707137040 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4abc0a3cd0 0x7f4abc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:19.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.791+0000 7f4abbfff700 1 -- 172.21.15.46:0/707137040 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f4ac80d1e40 con 0x7f4ad4103e60 2024-04-26T21:33:19.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.791+0000 7f4abbfff700 1 --2- 172.21.15.46:0/707137040 >> [v2:172.21.15.142:6816/1824918711,v1:172.21.15.142:6817/1824918711] conn(0x7f4abc0ad4b0 0x7f4abc0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:19.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.791+0000 7f4abbfff700 1 -- 172.21.15.46:0/707137040 --> [v2:172.21.15.142:6816/1824918711,v1:172.21.15.142:6817/1824918711] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4abc0aff80 con 0x7f4abc0ad4b0 2024-04-26T21:33:19.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.791+0000 7f4abbfff700 1 -- 172.21.15.46:0/707137040 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f4ac80d21f0 con 0x7f4ad4103e60 2024-04-26T21:33:19.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.792+0000 7f4ad2ffd700 1 --2- 172.21.15.46:0/707137040 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4abc0a3cd0 0x7f4abc0a6190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4ad419aaa0 tx=0x7f4ac400a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:19.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.792+0000 7f4ad37fe700 1 --2- 172.21.15.46:0/707137040 >> [v2:172.21.15.142:6816/1824918711,v1:172.21.15.142:6817/1824918711] conn(0x7f4abc0ad4b0 0x7f4abc0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:19.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.792+0000 7f4ad37fe700 1 --2- 172.21.15.46:0/707137040 >> [v2:172.21.15.142:6816/1824918711,v1:172.21.15.142:6817/1824918711] conn(0x7f4abc0ad4b0 0x7f4abc0af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:19.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.793+0000 7f4abbfff700 1 -- 172.21.15.46:0/707137040 <== osd.8 v2:172.21.15.142:6816/1824918711 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4abc0aff80 con 0x7f4abc0ad4b0 2024-04-26T21:33:19.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.809+0000 7f4ad92b6700 1 -- 172.21.15.46:0/707137040 --> [v2:172.21.15.142:6816/1824918711,v1:172.21.15.142:6817/1824918711] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4ad4063fb0 con 0x7f4abc0ad4b0 2024-04-26T21:33:19.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.810+0000 7f4abbfff700 1 -- 172.21.15.46:0/707137040 <== osd.8 v2:172.21.15.142:6816/1824918711 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4ad4063fb0 con 0x7f4abc0ad4b0 2024-04-26T21:33:19.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.810+0000 7f4ad92b6700 1 -- 172.21.15.46:0/707137040 >> [v2:172.21.15.142:6816/1824918711,v1:172.21.15.142:6817/1824918711] conn(0x7f4abc0ad4b0 msgr2=0x7f4abc0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.810+0000 7f4ad92b6700 1 --2- 172.21.15.46:0/707137040 >> [v2:172.21.15.142:6816/1824918711,v1:172.21.15.142:6817/1824918711] conn(0x7f4abc0ad4b0 0x7f4abc0af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.810+0000 7f4ad92b6700 1 -- 172.21.15.46:0/707137040 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4abc0a3cd0 msgr2=0x7f4abc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.810+0000 7f4ad92b6700 1 --2- 172.21.15.46:0/707137040 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4abc0a3cd0 0x7f4abc0a6190 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4ad419aaa0 tx=0x7f4ac400a380 comp rx=0 tx=0).stop 2024-04-26T21:33:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.810+0000 7f4ad92b6700 1 -- 172.21.15.46:0/707137040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ad4103e60 msgr2=0x7f4ad419fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.810+0000 7f4ad92b6700 1 --2- 172.21.15.46:0/707137040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ad4103e60 0x7f4ad419fce0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4ac80057f0 tx=0x7f4ac8003680 comp rx=0 tx=0).stop 2024-04-26T21:33:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.810+0000 7f4ad37fe700 1 -- 172.21.15.46:0/707137040 reap_dead start 2024-04-26T21:33:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.810+0000 7f4ad92b6700 1 -- 172.21.15.46:0/707137040 shutdown_connections 2024-04-26T21:33:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.810+0000 7f4ad92b6700 1 -- 172.21.15.46:0/707137040 >> 172.21.15.46:0/707137040 conn(0x7f4ad407a590 msgr2=0x7f4ad4102130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.811+0000 7f4ad92b6700 1 -- 172.21.15.46:0/707137040 shutdown_connections 2024-04-26T21:33:19.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:19.811+0000 7f4ad92b6700 1 -- 172.21.15.46:0/707137040 wait complete. 2024-04-26T21:33:20.667 INFO:teuthology.orchestra.run.smithi046.stdout:253403070500 2024-04-26T21:33:20.667 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-26T21:33:21.282 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:21 smithi046 ceph-mon[29201]: pgmap v310: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-26T21:33:21.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:21 smithi142 ceph-mon[32944]: pgmap v310: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-26T21:33:21.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:21 smithi050 ceph-mon[32753]: pgmap v310: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-04-26T21:33:22.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.735+0000 7fde9cb26700 1 -- 172.21.15.46:0/958108458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde981034a0 msgr2=0x7fde98103920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:22.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.735+0000 7fde9cb26700 1 --2- 172.21.15.46:0/958108458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde981034a0 0x7fde98103920 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fde8c009990 tx=0x7fde8c009ca0 comp rx=0 tx=0).stop 2024-04-26T21:33:22.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.735+0000 7fde9cb26700 1 -- 172.21.15.46:0/958108458 shutdown_connections 2024-04-26T21:33:22.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.735+0000 7fde9cb26700 1 --2- 172.21.15.46:0/958108458 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde98103e60 0x7fde981102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:22.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.735+0000 7fde9cb26700 1 --2- 172.21.15.46:0/958108458 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde981094e0 0x7fde981098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:22.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.735+0000 7fde9cb26700 1 --2- 172.21.15.46:0/958108458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde981034a0 0x7fde98103920 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:22.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.735+0000 7fde9cb26700 1 -- 172.21.15.46:0/958108458 >> 172.21.15.46:0/958108458 conn(0x7fde9807a590 msgr2=0x7fde9807a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:22.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.735+0000 7fde9cb26700 1 -- 172.21.15.46:0/958108458 shutdown_connections 2024-04-26T21:33:22.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.735+0000 7fde9cb26700 1 -- 172.21.15.46:0/958108458 wait complete. 2024-04-26T21:33:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.736+0000 7fde9cb26700 1 Processor -- start 2024-04-26T21:33:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.736+0000 7fde9cb26700 1 -- start start 2024-04-26T21:33:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.736+0000 7fde9cb26700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde981034a0 0x7fde9819f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.736+0000 7fde9cb26700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde98103e60 0x7fde9819fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.736+0000 7fde9cb26700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde981094e0 0x7fde981999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.736+0000 7fde9cb26700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fde98113610 con 0x7fde981094e0 2024-04-26T21:33:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.736+0000 7fde9cb26700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fde98113490 con 0x7fde98103e60 2024-04-26T21:33:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.736+0000 7fde9cb26700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fde98113790 con 0x7fde981034a0 2024-04-26T21:33:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.736+0000 7fde96d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde981094e0 0x7fde981999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.736+0000 7fde96d9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde981094e0 0x7fde981999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44784/0 (socket says 172.21.15.46:44784) 2024-04-26T21:33:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.736+0000 7fde96d9d700 1 -- 172.21.15.46:0/3422371289 learned_addr learned my addr 172.21.15.46:0/3422371289 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:33:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.736+0000 7fde9659c700 1 --2- 172.21.15.46:0/3422371289 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde981034a0 0x7fde9819f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.736+0000 7fde95d9b700 1 --2- 172.21.15.46:0/3422371289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde98103e60 0x7fde9819fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.737+0000 7fde96d9d700 1 -- 172.21.15.46:0/3422371289 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde981034a0 msgr2=0x7fde9819f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.737+0000 7fde96d9d700 1 --2- 172.21.15.46:0/3422371289 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde981034a0 0x7fde9819f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.737+0000 7fde96d9d700 1 -- 172.21.15.46:0/3422371289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde98103e60 msgr2=0x7fde9819fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.737+0000 7fde96d9d700 1 --2- 172.21.15.46:0/3422371289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde98103e60 0x7fde9819fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.737+0000 7fde96d9d700 1 -- 172.21.15.46:0/3422371289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde9819a110 con 0x7fde981094e0 2024-04-26T21:33:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.737+0000 7fde96d9d700 1 --2- 172.21.15.46:0/3422371289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde981094e0 0x7fde981999b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fde90007ee0 tx=0x7fde9000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.737+0000 7fde877fe700 1 -- 172.21.15.46:0/3422371289 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde90004bb0 con 0x7fde981094e0 2024-04-26T21:33:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.737+0000 7fde877fe700 1 -- 172.21.15.46:0/3422371289 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde90004d50 con 0x7fde981094e0 2024-04-26T21:33:22.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.737+0000 7fde9cb26700 1 -- 172.21.15.46:0/3422371289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde9819a400 con 0x7fde981094e0 2024-04-26T21:33:22.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.737+0000 7fde877fe700 1 -- 172.21.15.46:0/3422371289 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde90005920 con 0x7fde981094e0 2024-04-26T21:33:22.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.737+0000 7fde9cb26700 1 -- 172.21.15.46:0/3422371289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde981a66c0 con 0x7fde981094e0 2024-04-26T21:33:22.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.738+0000 7fde9cb26700 1 -- 172.21.15.46:0/3422371289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fde98063af0 con 0x7fde981094e0 2024-04-26T21:33:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.740+0000 7fde877fe700 1 -- 172.21.15.46:0/3422371289 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fde9001a020 con 0x7fde981094e0 2024-04-26T21:33:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.740+0000 7fde877fe700 1 --2- 172.21.15.46:0/3422371289 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fde800a3cd0 0x7fde800a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.740+0000 7fde9659c700 1 --2- 172.21.15.46:0/3422371289 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fde800a3cd0 0x7fde800a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.740+0000 7fde877fe700 1 -- 172.21.15.46:0/3422371289 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fde900d0130 con 0x7fde981094e0 2024-04-26T21:33:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.740+0000 7fde877fe700 1 --2- 172.21.15.46:0/3422371289 >> [v2:172.21.15.50:6816/820310228,v1:172.21.15.50:6817/820310228] conn(0x7fde800ad4b0 0x7fde800af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.740+0000 7fde877fe700 1 -- 172.21.15.46:0/3422371289 --> [v2:172.21.15.50:6816/820310228,v1:172.21.15.50:6817/820310228] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fde800aff80 con 0x7fde800ad4b0 2024-04-26T21:33:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.740+0000 7fde877fe700 1 -- 172.21.15.46:0/3422371289 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fde900d0610 con 0x7fde981094e0 2024-04-26T21:33:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.741+0000 7fde9659c700 1 --2- 172.21.15.46:0/3422371289 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fde800a3cd0 0x7fde800a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fde9819aaf0 tx=0x7fde88009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.741+0000 7fde95d9b700 1 --2- 172.21.15.46:0/3422371289 >> [v2:172.21.15.50:6816/820310228,v1:172.21.15.50:6817/820310228] conn(0x7fde800ad4b0 0x7fde800af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.741+0000 7fde95d9b700 1 --2- 172.21.15.46:0/3422371289 >> [v2:172.21.15.50:6816/820310228,v1:172.21.15.50:6817/820310228] conn(0x7fde800ad4b0 0x7fde800af8d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:22.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.742+0000 7fde877fe700 1 -- 172.21.15.46:0/3422371289 <== osd.7 v2:172.21.15.50:6816/820310228 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fde800aff80 con 0x7fde800ad4b0 2024-04-26T21:33:22.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.758+0000 7fde9cb26700 1 -- 172.21.15.46:0/3422371289 --> [v2:172.21.15.50:6816/820310228,v1:172.21.15.50:6817/820310228] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fde98063f70 con 0x7fde800ad4b0 2024-04-26T21:33:22.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.759+0000 7fde877fe700 1 -- 172.21.15.46:0/3422371289 <== osd.7 v2:172.21.15.50:6816/820310228 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fde98063f70 con 0x7fde800ad4b0 2024-04-26T21:33:22.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.759+0000 7fde9cb26700 1 -- 172.21.15.46:0/3422371289 >> [v2:172.21.15.50:6816/820310228,v1:172.21.15.50:6817/820310228] conn(0x7fde800ad4b0 msgr2=0x7fde800af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:22.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.759+0000 7fde9cb26700 1 --2- 172.21.15.46:0/3422371289 >> [v2:172.21.15.50:6816/820310228,v1:172.21.15.50:6817/820310228] conn(0x7fde800ad4b0 0x7fde800af8d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:22.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.759+0000 7fde9cb26700 1 -- 172.21.15.46:0/3422371289 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fde800a3cd0 msgr2=0x7fde800a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:22.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.759+0000 7fde9cb26700 1 --2- 172.21.15.46:0/3422371289 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fde800a3cd0 0x7fde800a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fde9819aaf0 tx=0x7fde88009260 comp rx=0 tx=0).stop 2024-04-26T21:33:22.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.759+0000 7fde9cb26700 1 -- 172.21.15.46:0/3422371289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde981094e0 msgr2=0x7fde981999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:22.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.759+0000 7fde9cb26700 1 --2- 172.21.15.46:0/3422371289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde981094e0 0x7fde981999b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fde90007ee0 tx=0x7fde9000d3b0 comp rx=0 tx=0).stop 2024-04-26T21:33:22.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.759+0000 7fde96d9d700 1 -- 172.21.15.46:0/3422371289 reap_dead start 2024-04-26T21:33:22.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.759+0000 7fde9cb26700 1 -- 172.21.15.46:0/3422371289 shutdown_connections 2024-04-26T21:33:22.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.759+0000 7fde9cb26700 1 -- 172.21.15.46:0/3422371289 >> 172.21.15.46:0/3422371289 conn(0x7fde9807a590 msgr2=0x7fde98102240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:22.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.760+0000 7fde9cb26700 1 -- 172.21.15.46:0/3422371289 shutdown_connections 2024-04-26T21:33:22.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:22.760+0000 7fde9cb26700 1 -- 172.21.15.46:0/3422371289 wait complete. 2024-04-26T21:33:23.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:23 smithi046 ceph-mon[29201]: pgmap v311: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T21:33:23.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:23 smithi142 ceph-mon[32944]: pgmap v311: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T21:33:23.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:23 smithi050 ceph-mon[32753]: pgmap v311: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T21:33:23.790 INFO:teuthology.orchestra.run.smithi046.stdout:343597383701 2024-04-26T21:33:23.790 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-26T21:33:24.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.394+0000 7f31eedae700 1 -- 172.21.15.46:0/453326916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31e8102f60 msgr2=0x7f31e8106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:24.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.394+0000 7f31eedae700 1 --2- 172.21.15.46:0/453326916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31e8102f60 0x7f31e8106f60 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f31dc0099c0 tx=0x7f31dc009cd0 comp rx=0 tx=0).stop 2024-04-26T21:33:24.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.394+0000 7f31eedae700 1 -- 172.21.15.46:0/453326916 shutdown_connections 2024-04-26T21:33:24.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.394+0000 7f31eedae700 1 --2- 172.21.15.46:0/453326916 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31e8107640 0x7f31e810f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:24.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.394+0000 7f31eedae700 1 --2- 172.21.15.46:0/453326916 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31e81025b0 0x7f31e8102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:24.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.394+0000 7f31eedae700 1 --2- 172.21.15.46:0/453326916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31e8102f60 0x7f31e8106f60 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:24.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.394+0000 7f31eedae700 1 -- 172.21.15.46:0/453326916 >> 172.21.15.46:0/453326916 conn(0x7f31e80792d0 msgr2=0x7f31e80796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:24.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.394+0000 7f31eedae700 1 -- 172.21.15.46:0/453326916 shutdown_connections 2024-04-26T21:33:24.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.394+0000 7f31eedae700 1 -- 172.21.15.46:0/453326916 wait complete. 2024-04-26T21:33:24.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.395+0000 7f31eedae700 1 Processor -- start 2024-04-26T21:33:24.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.395+0000 7f31eedae700 1 -- start start 2024-04-26T21:33:24.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.395+0000 7f31eedae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31e81025b0 0x7f31e819f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:24.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.395+0000 7f31eedae700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31e8102f60 0x7f31e819fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:24.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.395+0000 7f31eedae700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31e8107640 0x7f31e8199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:24.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.395+0000 7f31eedae700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f31e81124d0 con 0x7f31e8102f60 2024-04-26T21:33:24.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.395+0000 7f31eedae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f31e8112350 con 0x7f31e81025b0 2024-04-26T21:33:24.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.395+0000 7f31eedae700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f31e8112650 con 0x7f31e8107640 2024-04-26T21:33:24.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.395+0000 7f31e7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31e8102f60 0x7f31e819fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:24.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.395+0000 7f31e7fff700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31e8102f60 0x7f31e819fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44822/0 (socket says 172.21.15.46:44822) 2024-04-26T21:33:24.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.395+0000 7f31e7fff700 1 -- 172.21.15.46:0/3582551286 learned_addr learned my addr 172.21.15.46:0/3582551286 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:33:24.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.395+0000 7f31ecb4a700 1 --2- 172.21.15.46:0/3582551286 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31e81025b0 0x7f31e819f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:24.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.395+0000 7f31ed34b700 1 --2- 172.21.15.46:0/3582551286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31e8107640 0x7f31e8199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:24.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.395+0000 7f31e7fff700 1 -- 172.21.15.46:0/3582551286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31e8107640 msgr2=0x7f31e8199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:24.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.395+0000 7f31e7fff700 1 --2- 172.21.15.46:0/3582551286 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31e8107640 0x7f31e8199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:24.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.395+0000 7f31e7fff700 1 -- 172.21.15.46:0/3582551286 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31e81025b0 msgr2=0x7f31e819f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:24.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.395+0000 7f31e7fff700 1 --2- 172.21.15.46:0/3582551286 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31e81025b0 0x7f31e819f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:24.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.395+0000 7f31e7fff700 1 -- 172.21.15.46:0/3582551286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31e819a0e0 con 0x7f31e8102f60 2024-04-26T21:33:24.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.395+0000 7f31ecb4a700 1 --2- 172.21.15.46:0/3582551286 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31e81025b0 0x7f31e819f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:33:24.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.395+0000 7f31e7fff700 1 --2- 172.21.15.46:0/3582551286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31e8102f60 0x7f31e819fd00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f31dc006010 tx=0x7f31dc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:24.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.396+0000 7f31e5ffb700 1 -- 172.21.15.46:0/3582551286 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31dc027070 con 0x7f31e8102f60 2024-04-26T21:33:24.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.396+0000 7f31eedae700 1 -- 172.21.15.46:0/3582551286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31e819a370 con 0x7f31e8102f60 2024-04-26T21:33:24.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.396+0000 7f31e5ffb700 1 -- 172.21.15.46:0/3582551286 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31dc003f20 con 0x7f31e8102f60 2024-04-26T21:33:24.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.396+0000 7f31e5ffb700 1 -- 172.21.15.46:0/3582551286 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31dc022cd0 con 0x7f31e8102f60 2024-04-26T21:33:24.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.396+0000 7f31eedae700 1 -- 172.21.15.46:0/3582551286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31e8103f30 con 0x7f31e8102f60 2024-04-26T21:33:24.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.397+0000 7f31eedae700 1 -- 172.21.15.46:0/3582551286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f31e8063bb0 con 0x7f31e8102f60 2024-04-26T21:33:24.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.398+0000 7f31e5ffb700 1 -- 172.21.15.46:0/3582551286 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f31dc02a070 con 0x7f31e8102f60 2024-04-26T21:33:24.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.399+0000 7f31e5ffb700 1 --2- 172.21.15.46:0/3582551286 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f31d00a3cd0 0x7f31d00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:24.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.399+0000 7f31ecb4a700 1 --2- 172.21.15.46:0/3582551286 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f31d00a3cd0 0x7f31d00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:24.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.399+0000 7f31e5ffb700 1 -- 172.21.15.46:0/3582551286 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f31dc0d1aa0 con 0x7f31e8102f60 2024-04-26T21:33:24.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.399+0000 7f31e5ffb700 1 --2- 172.21.15.46:0/3582551286 >> [v2:172.21.15.46:6818/2325162882,v1:172.21.15.46:6819/2325162882] conn(0x7f31d00ad4b0 0x7f31d00af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:24.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.399+0000 7f31e5ffb700 1 -- 172.21.15.46:0/3582551286 --> [v2:172.21.15.46:6818/2325162882,v1:172.21.15.46:6819/2325162882] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f31dc0043d0 con 0x7f31d00ad4b0 2024-04-26T21:33:24.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.399+0000 7f31e5ffb700 1 -- 172.21.15.46:0/3582551286 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f31dc0d1e50 con 0x7f31e8102f60 2024-04-26T21:33:24.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.399+0000 7f31ed34b700 1 --2- 172.21.15.46:0/3582551286 >> [v2:172.21.15.46:6818/2325162882,v1:172.21.15.46:6819/2325162882] conn(0x7f31d00ad4b0 0x7f31d00af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:24.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.399+0000 7f31ecb4a700 1 --2- 172.21.15.46:0/3582551286 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f31d00a3cd0 0x7f31d00a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f31d8005950 tx=0x7f31d80058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:24.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.399+0000 7f31ed34b700 1 --2- 172.21.15.46:0/3582551286 >> [v2:172.21.15.46:6818/2325162882,v1:172.21.15.46:6819/2325162882] conn(0x7f31d00ad4b0 0x7f31d00af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:24.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.400+0000 7f31e5ffb700 1 -- 172.21.15.46:0/3582551286 <== osd.6 v2:172.21.15.46:6818/2325162882 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f31dc0043d0 con 0x7f31d00ad4b0 2024-04-26T21:33:24.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.416+0000 7f31eedae700 1 -- 172.21.15.46:0/3582551286 --> [v2:172.21.15.46:6818/2325162882,v1:172.21.15.46:6819/2325162882] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f31e8063fb0 con 0x7f31d00ad4b0 2024-04-26T21:33:24.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.417+0000 7f31e5ffb700 1 -- 172.21.15.46:0/3582551286 <== osd.6 v2:172.21.15.46:6818/2325162882 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f31e8063fb0 con 0x7f31d00ad4b0 2024-04-26T21:33:24.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.417+0000 7f31eedae700 1 -- 172.21.15.46:0/3582551286 >> [v2:172.21.15.46:6818/2325162882,v1:172.21.15.46:6819/2325162882] conn(0x7f31d00ad4b0 msgr2=0x7f31d00af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:24.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.417+0000 7f31eedae700 1 --2- 172.21.15.46:0/3582551286 >> [v2:172.21.15.46:6818/2325162882,v1:172.21.15.46:6819/2325162882] conn(0x7f31d00ad4b0 0x7f31d00af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:24.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.417+0000 7f31eedae700 1 -- 172.21.15.46:0/3582551286 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f31d00a3cd0 msgr2=0x7f31d00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:24.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.417+0000 7f31eedae700 1 --2- 172.21.15.46:0/3582551286 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f31d00a3cd0 0x7f31d00a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f31d8005950 tx=0x7f31d80058e0 comp rx=0 tx=0).stop 2024-04-26T21:33:24.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.417+0000 7f31eedae700 1 -- 172.21.15.46:0/3582551286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31e8102f60 msgr2=0x7f31e819fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:24.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.417+0000 7f31eedae700 1 --2- 172.21.15.46:0/3582551286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31e8102f60 0x7f31e819fd00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f31dc006010 tx=0x7f31dc0047b0 comp rx=0 tx=0).stop 2024-04-26T21:33:24.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.417+0000 7f31ed34b700 1 -- 172.21.15.46:0/3582551286 reap_dead start 2024-04-26T21:33:24.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.417+0000 7f31eedae700 1 -- 172.21.15.46:0/3582551286 shutdown_connections 2024-04-26T21:33:24.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.417+0000 7f31eedae700 1 -- 172.21.15.46:0/3582551286 >> 172.21.15.46:0/3582551286 conn(0x7f31e80792d0 msgr2=0x7f31e806bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:24.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.417+0000 7f31eedae700 1 -- 172.21.15.46:0/3582551286 shutdown_connections 2024-04-26T21:33:24.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:24.417+0000 7f31eedae700 1 -- 172.21.15.46:0/3582551286 wait complete. 2024-04-26T21:33:24.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.065+0000 7fd338814700 1 -- 172.21.15.46:0/3069463465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3341025b0 msgr2=0x7fd334102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:24.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.065+0000 7fd338814700 1 --2- 172.21.15.46:0/3069463465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3341025b0 0x7fd334102990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd3240099c0 tx=0x7fd324009cd0 comp rx=0 tx=0).stop 2024-04-26T21:33:24.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.065+0000 7fd338814700 1 -- 172.21.15.46:0/3069463465 shutdown_connections 2024-04-26T21:33:24.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.065+0000 7fd338814700 1 --2- 172.21.15.46:0/3069463465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3341025b0 0x7fd334102990 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:24.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.065+0000 7fd338814700 1 --2- 172.21.15.46:0/3069463465 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd334107640 0x7fd33410f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:24.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.065+0000 7fd338814700 1 --2- 172.21.15.46:0/3069463465 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd334102f60 0x7fd334106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:24.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.065+0000 7fd338814700 1 -- 172.21.15.46:0/3069463465 >> 172.21.15.46:0/3069463465 conn(0x7fd3340792d0 msgr2=0x7fd3340796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:24.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.065+0000 7fd338814700 1 -- 172.21.15.46:0/3069463465 shutdown_connections 2024-04-26T21:33:24.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.065+0000 7fd338814700 1 -- 172.21.15.46:0/3069463465 wait complete. 2024-04-26T21:33:24.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.066+0000 7fd338814700 1 Processor -- start 2024-04-26T21:33:24.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.066+0000 7fd338814700 1 -- start start 2024-04-26T21:33:24.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.066+0000 7fd338814700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3341025b0 0x7fd33419e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:24.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.066+0000 7fd338814700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd334102f60 0x7fd33419ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:24.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.066+0000 7fd338814700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd334107640 0x7fd3341987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:24.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.066+0000 7fd338814700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fd3341124f0 con 0x7fd3341025b0 2024-04-26T21:33:24.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.066+0000 7fd338814700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd334112370 con 0x7fd334107640 2024-04-26T21:33:24.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.066+0000 7fd338814700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fd334112670 con 0x7fd334102f60 2024-04-26T21:33:24.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.066+0000 7fd33259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3341025b0 0x7fd33419e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:24.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.066+0000 7fd33259c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3341025b0 0x7fd33419e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44808/0 (socket says 172.21.15.46:44808) 2024-04-26T21:33:24.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.066+0000 7fd33259c700 1 -- 172.21.15.46:0/794449197 learned_addr learned my addr 172.21.15.46:0/794449197 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:33:24.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.066+0000 7fd331d9b700 1 --2- 172.21.15.46:0/794449197 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd334102f60 0x7fd33419ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:24.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.066+0000 7fd332d9d700 1 --2- 172.21.15.46:0/794449197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd334107640 0x7fd3341987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:24.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.067+0000 7fd33259c700 1 -- 172.21.15.46:0/794449197 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd334102f60 msgr2=0x7fd33419ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:24.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.067+0000 7fd33259c700 1 --2- 172.21.15.46:0/794449197 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fd334102f60 0x7fd33419ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:24.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.067+0000 7fd33259c700 1 -- 172.21.15.46:0/794449197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd334107640 msgr2=0x7fd3341987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:24.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.067+0000 7fd33259c700 1 --2- 172.21.15.46:0/794449197 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd334107640 0x7fd3341987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:24.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.067+0000 7fd33259c700 1 -- 172.21.15.46:0/794449197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd334198fd0 con 0x7fd3341025b0 2024-04-26T21:33:24.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.067+0000 7fd33259c700 1 --2- 172.21.15.46:0/794449197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3341025b0 0x7fd33419e670 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd32400b580 tx=0x7fd3240047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:24.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.067+0000 7fd3237fe700 1 -- 172.21.15.46:0/794449197 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd324027070 con 0x7fd3341025b0 2024-04-26T21:33:24.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.067+0000 7fd338814700 1 -- 172.21.15.46:0/794449197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd334199260 con 0x7fd3341025b0 2024-04-26T21:33:24.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.067+0000 7fd3237fe700 1 -- 172.21.15.46:0/794449197 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd324003f20 con 0x7fd3341025b0 2024-04-26T21:33:24.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.067+0000 7fd3237fe700 1 -- 172.21.15.46:0/794449197 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd324022d70 con 0x7fd3341025b0 2024-04-26T21:33:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.067+0000 7fd338814700 1 -- 172.21.15.46:0/794449197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3341a54d0 con 0x7fd3341025b0 2024-04-26T21:33:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.068+0000 7fd338814700 1 -- 172.21.15.46:0/794449197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd334063bb0 con 0x7fd3341025b0 2024-04-26T21:33:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.069+0000 7fd3237fe700 1 -- 172.21.15.46:0/794449197 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd324022500 con 0x7fd3341025b0 2024-04-26T21:33:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.070+0000 7fd3237fe700 1 --2- 172.21.15.46:0/794449197 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fd31c0a3cd0 0x7fd31c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.070+0000 7fd331d9b700 1 --2- 172.21.15.46:0/794449197 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fd31c0a3cd0 0x7fd31c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.070+0000 7fd3237fe700 1 -- 172.21.15.46:0/794449197 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fd3240d17b0 con 0x7fd3341025b0 2024-04-26T21:33:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.070+0000 7fd3237fe700 1 --2- 172.21.15.46:0/794449197 >> [v2:172.21.15.46:6826/3601947736,v1:172.21.15.46:6827/3601947736] conn(0x7fd31c0ad380 0x7fd31c0af7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.070+0000 7fd3237fe700 1 -- 172.21.15.46:0/794449197 --> [v2:172.21.15.46:6826/3601947736,v1:172.21.15.46:6827/3601947736] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd324019e80 con 0x7fd31c0ad380 2024-04-26T21:33:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.070+0000 7fd3237fe700 1 -- 172.21.15.46:0/794449197 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fd3240d1b60 con 0x7fd3341025b0 2024-04-26T21:33:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.070+0000 7fd332d9d700 1 --2- 172.21.15.46:0/794449197 >> [v2:172.21.15.46:6826/3601947736,v1:172.21.15.46:6827/3601947736] conn(0x7fd31c0ad380 0x7fd31c0af7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.070+0000 7fd331d9b700 1 --2- 172.21.15.46:0/794449197 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fd31c0a3cd0 0x7fd31c0a6190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd334101f80 tx=0x7fd32800b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.071+0000 7fd332d9d700 1 --2- 172.21.15.46:0/794449197 >> [v2:172.21.15.46:6826/3601947736,v1:172.21.15.46:6827/3601947736] conn(0x7fd31c0ad380 0x7fd31c0af7a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:24.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.071+0000 7fd3237fe700 1 -- 172.21.15.46:0/794449197 <== osd.9 v2:172.21.15.46:6826/3601947736 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd324019e80 con 0x7fd31c0ad380 2024-04-26T21:33:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.088+0000 7fd338814700 1 -- 172.21.15.46:0/794449197 --> [v2:172.21.15.46:6826/3601947736,v1:172.21.15.46:6827/3601947736] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd334063fb0 con 0x7fd31c0ad380 2024-04-26T21:33:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.088+0000 7fd3237fe700 1 -- 172.21.15.46:0/794449197 <== osd.9 v2:172.21.15.46:6826/3601947736 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd334063fb0 con 0x7fd31c0ad380 2024-04-26T21:33:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.088+0000 7fd338814700 1 -- 172.21.15.46:0/794449197 >> [v2:172.21.15.46:6826/3601947736,v1:172.21.15.46:6827/3601947736] conn(0x7fd31c0ad380 msgr2=0x7fd31c0af7a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.088+0000 7fd338814700 1 --2- 172.21.15.46:0/794449197 >> [v2:172.21.15.46:6826/3601947736,v1:172.21.15.46:6827/3601947736] conn(0x7fd31c0ad380 0x7fd31c0af7a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.088+0000 7fd338814700 1 -- 172.21.15.46:0/794449197 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fd31c0a3cd0 msgr2=0x7fd31c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.088+0000 7fd338814700 1 --2- 172.21.15.46:0/794449197 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fd31c0a3cd0 0x7fd31c0a6190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd334101f80 tx=0x7fd32800b3c0 comp rx=0 tx=0).stop 2024-04-26T21:33:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.088+0000 7fd338814700 1 -- 172.21.15.46:0/794449197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3341025b0 msgr2=0x7fd33419e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.088+0000 7fd338814700 1 --2- 172.21.15.46:0/794449197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3341025b0 0x7fd33419e670 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd32400b580 tx=0x7fd3240047b0 comp rx=0 tx=0).stop 2024-04-26T21:33:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.088+0000 7fd332d9d700 1 -- 172.21.15.46:0/794449197 reap_dead start 2024-04-26T21:33:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.089+0000 7fd338814700 1 -- 172.21.15.46:0/794449197 shutdown_connections 2024-04-26T21:33:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.089+0000 7fd338814700 1 -- 172.21.15.46:0/794449197 >> 172.21.15.46:0/794449197 conn(0x7fd3340792d0 msgr2=0x7fd3341067d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.089+0000 7fd338814700 1 -- 172.21.15.46:0/794449197 shutdown_connections 2024-04-26T21:33:24.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:23.089+0000 7fd338814700 1 -- 172.21.15.46:0/794449197 wait complete. 2024-04-26T21:33:25.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:25 smithi142 ceph-mon[32944]: pgmap v312: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T21:33:25.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:25 smithi046 ceph-mon[29201]: pgmap v312: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T21:33:25.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:25 smithi050 ceph-mon[32753]: pgmap v312: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-26T21:33:26.560 INFO:teuthology.orchestra.run.smithi046.stdout:287762808864 2024-04-26T21:33:26.561 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-26T21:33:27.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:27 smithi046 ceph-mon[29201]: pgmap v313: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-26T21:33:27.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:27 smithi142 ceph-mon[32944]: pgmap v313: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-26T21:33:27.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:27 smithi050 ceph-mon[32753]: pgmap v313: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-26T21:33:27.852 INFO:teuthology.orchestra.run.smithi046.stdout:219043332138 2024-04-26T21:33:27.852 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-26T21:33:28.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.070+0000 7f4209884700 1 -- 172.21.15.46:0/763519617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4204108390 msgr2=0x7f4204108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:28.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.070+0000 7f4209884700 1 --2- 172.21.15.46:0/763519617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4204108390 0x7f4204108770 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f41f4009990 tx=0x7f41f4009ca0 comp rx=0 tx=0).stop 2024-04-26T21:33:28.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.070+0000 7f4209884700 1 -- 172.21.15.46:0/763519617 shutdown_connections 2024-04-26T21:33:28.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.070+0000 7f4209884700 1 --2- 172.21.15.46:0/763519617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4204108390 0x7f4204108770 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:28.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.071+0000 7f4209884700 1 --2- 172.21.15.46:0/763519617 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4204102ce0 0x7f420410f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:28.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.071+0000 7f4209884700 1 --2- 172.21.15.46:0/763519617 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4204102320 0x7f42041027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:28.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.071+0000 7f4209884700 1 -- 172.21.15.46:0/763519617 >> 172.21.15.46:0/763519617 conn(0x7f42040792d0 msgr2=0x7f42040796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:28.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.071+0000 7f4209884700 1 -- 172.21.15.46:0/763519617 shutdown_connections 2024-04-26T21:33:28.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.071+0000 7f4209884700 1 -- 172.21.15.46:0/763519617 wait complete. 2024-04-26T21:33:28.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.071+0000 7f4209884700 1 Processor -- start 2024-04-26T21:33:28.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.072+0000 7f4209884700 1 -- start start 2024-04-26T21:33:28.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.072+0000 7f4209884700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4204102320 0x7f4204104000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:28.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.072+0000 7f4209884700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4204102ce0 0x7f4204104540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:28.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.072+0000 7f4209884700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4204108390 0x7f4204104a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:28.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.072+0000 7f4209884700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4204112500 con 0x7f4204102320 2024-04-26T21:33:28.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.072+0000 7f4209884700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4204112380 con 0x7f4204102ce0 2024-04-26T21:33:28.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.072+0000 7f4209884700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4204112680 con 0x7f4204108390 2024-04-26T21:33:28.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.072+0000 7f4202ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4204102320 0x7f4204104000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:28.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.072+0000 7f4202ffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4204102320 0x7f4204104000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47604/0 (socket says 172.21.15.46:47604) 2024-04-26T21:33:28.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.072+0000 7f4202ffd700 1 -- 172.21.15.46:0/2133638907 learned_addr learned my addr 172.21.15.46:0/2133638907 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:33:28.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.072+0000 7f42027fc700 1 --2- 172.21.15.46:0/2133638907 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4204102ce0 0x7f4204104540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:28.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.072+0000 7f42037fe700 1 --2- 172.21.15.46:0/2133638907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4204108390 0x7f4204104a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:28.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.073+0000 7f4202ffd700 1 -- 172.21.15.46:0/2133638907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4204108390 msgr2=0x7f4204104a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:28.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.073+0000 7f4202ffd700 1 --2- 172.21.15.46:0/2133638907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4204108390 0x7f4204104a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:28.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.073+0000 7f4202ffd700 1 -- 172.21.15.46:0/2133638907 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4204102ce0 msgr2=0x7f4204104540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:28.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.073+0000 7f4202ffd700 1 --2- 172.21.15.46:0/2133638907 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4204102ce0 0x7f4204104540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:28.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.073+0000 7f4202ffd700 1 -- 172.21.15.46:0/2133638907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42041a8070 con 0x7f4204102320 2024-04-26T21:33:28.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.073+0000 7f42027fc700 1 --2- 172.21.15.46:0/2133638907 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4204102ce0 0x7f4204104540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:33:28.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.073+0000 7f42037fe700 1 --2- 172.21.15.46:0/2133638907 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4204108390 0x7f4204104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:33:28.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.073+0000 7f4202ffd700 1 --2- 172.21.15.46:0/2133638907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4204102320 0x7f4204104000 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f41f40047e0 tx=0x7f41f4009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:28.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.073+0000 7f4208882700 1 -- 172.21.15.46:0/2133638907 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4026070 con 0x7f4204102320 2024-04-26T21:33:28.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.073+0000 7f4209884700 1 -- 172.21.15.46:0/2133638907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42041a8300 con 0x7f4204102320 2024-04-26T21:33:28.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.073+0000 7f4209884700 1 -- 172.21.15.46:0/2133638907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42041a87b0 con 0x7f4204102320 2024-04-26T21:33:28.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.073+0000 7f4208882700 1 -- 172.21.15.46:0/2133638907 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41f4004cc0 con 0x7f4204102320 2024-04-26T21:33:28.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.073+0000 7f4208882700 1 -- 172.21.15.46:0/2133638907 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f4003cd0 con 0x7f4204102320 2024-04-26T21:33:28.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.074+0000 7f4209884700 1 -- 172.21.15.46:0/2133638907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f41f0001030 con 0x7f4204102320 2024-04-26T21:33:28.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.075+0000 7f4208882700 1 -- 172.21.15.46:0/2133638907 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f41f4029070 con 0x7f4204102320 2024-04-26T21:33:28.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.076+0000 7f4208882700 1 --2- 172.21.15.46:0/2133638907 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f41ec0a3d50 0x7f41ec0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:28.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.076+0000 7f4208882700 1 -- 172.21.15.46:0/2133638907 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f41f40d12e0 con 0x7f4204102320 2024-04-26T21:33:28.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.076+0000 7f42027fc700 1 --2- 172.21.15.46:0/2133638907 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f41ec0a3d50 0x7f41ec0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:28.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.076+0000 7f4208882700 1 --2- 172.21.15.46:0/2133638907 >> [v2:172.21.15.50:6800/1628008970,v1:172.21.15.50:6801/1628008970] conn(0x7f41ec0ad530 0x7f41ec0af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:28.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.076+0000 7f4208882700 1 -- 172.21.15.46:0/2133638907 --> [v2:172.21.15.50:6800/1628008970,v1:172.21.15.50:6801/1628008970] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f41ec0b0000 con 0x7f41ec0ad530 2024-04-26T21:33:28.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.076+0000 7f4208882700 1 -- 172.21.15.46:0/2133638907 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f41f40d17c0 con 0x7f4204102320 2024-04-26T21:33:28.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.076+0000 7f42027fc700 1 --2- 172.21.15.46:0/2133638907 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f41ec0a3d50 0x7f41ec0a6210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f41f8005950 tx=0x7f41f80058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:28.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.077+0000 7f42037fe700 1 --2- 172.21.15.46:0/2133638907 >> [v2:172.21.15.50:6800/1628008970,v1:172.21.15.50:6801/1628008970] conn(0x7f41ec0ad530 0x7f41ec0af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:28.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.077+0000 7f42037fe700 1 --2- 172.21.15.46:0/2133638907 >> [v2:172.21.15.50:6800/1628008970,v1:172.21.15.50:6801/1628008970] conn(0x7f41ec0ad530 0x7f41ec0af950 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:28.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.078+0000 7f4208882700 1 -- 172.21.15.46:0/2133638907 <== osd.1 v2:172.21.15.50:6800/1628008970 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f41ec0b0000 con 0x7f41ec0ad530 2024-04-26T21:33:28.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.094+0000 7f4209884700 1 -- 172.21.15.46:0/2133638907 --> [v2:172.21.15.50:6800/1628008970,v1:172.21.15.50:6801/1628008970] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f41f0002e60 con 0x7f41ec0ad530 2024-04-26T21:33:28.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.095+0000 7f4208882700 1 -- 172.21.15.46:0/2133638907 <== osd.1 v2:172.21.15.50:6800/1628008970 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f41f0002e60 con 0x7f41ec0ad530 2024-04-26T21:33:28.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.095+0000 7f4209884700 1 -- 172.21.15.46:0/2133638907 >> [v2:172.21.15.50:6800/1628008970,v1:172.21.15.50:6801/1628008970] conn(0x7f41ec0ad530 msgr2=0x7f41ec0af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:28.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.095+0000 7f4209884700 1 --2- 172.21.15.46:0/2133638907 >> [v2:172.21.15.50:6800/1628008970,v1:172.21.15.50:6801/1628008970] conn(0x7f41ec0ad530 0x7f41ec0af950 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:28.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.095+0000 7f4209884700 1 -- 172.21.15.46:0/2133638907 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f41ec0a3d50 msgr2=0x7f41ec0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:28.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.095+0000 7f4209884700 1 --2- 172.21.15.46:0/2133638907 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f41ec0a3d50 0x7f41ec0a6210 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f41f8005950 tx=0x7f41f80058e0 comp rx=0 tx=0).stop 2024-04-26T21:33:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.095+0000 7f4209884700 1 -- 172.21.15.46:0/2133638907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4204102320 msgr2=0x7f4204104000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.095+0000 7f4209884700 1 --2- 172.21.15.46:0/2133638907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4204102320 0x7f4204104000 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f41f40047e0 tx=0x7f41f4009ee0 comp rx=0 tx=0).stop 2024-04-26T21:33:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.095+0000 7f42037fe700 1 -- 172.21.15.46:0/2133638907 reap_dead start 2024-04-26T21:33:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.095+0000 7f4209884700 1 -- 172.21.15.46:0/2133638907 shutdown_connections 2024-04-26T21:33:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.095+0000 7f4209884700 1 -- 172.21.15.46:0/2133638907 >> 172.21.15.46:0/2133638907 conn(0x7f42040792d0 msgr2=0x7f420406b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.095+0000 7f4209884700 1 -- 172.21.15.46:0/2133638907 shutdown_connections 2024-04-26T21:33:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.095+0000 7f4209884700 1 -- 172.21.15.46:0/2133638907 wait complete. 2024-04-26T21:33:28.770 INFO:teuthology.orchestra.run.smithi046.stdout:141733920825 2024-04-26T21:33:28.771 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-26T21:33:28.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.042+0000 7f7c2de4f700 1 -- 172.21.15.46:0/3961942546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c28109490 msgr2=0x7f7c28109870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:28.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.042+0000 7f7c2de4f700 1 --2- 172.21.15.46:0/3961942546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c28109490 0x7f7c28109870 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7c18009990 tx=0x7f7c18009ca0 comp rx=0 tx=0).stop 2024-04-26T21:33:28.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.042+0000 7f7c2de4f700 1 -- 172.21.15.46:0/3961942546 shutdown_connections 2024-04-26T21:33:28.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.042+0000 7f7c2de4f700 1 --2- 172.21.15.46:0/3961942546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c28109490 0x7f7c28109870 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:28.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.042+0000 7f7c2de4f700 1 --2- 172.21.15.46:0/3961942546 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c2806c360 0x7f7c281102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:28.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.042+0000 7f7c2de4f700 1 --2- 172.21.15.46:0/3961942546 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c2806b9c0 0x7f7c2806be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:28.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.042+0000 7f7c2de4f700 1 -- 172.21.15.46:0/3961942546 >> 172.21.15.46:0/3961942546 conn(0x7f7c28079290 msgr2=0x7f7c280796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:28.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.042+0000 7f7c2de4f700 1 -- 172.21.15.46:0/3961942546 shutdown_connections 2024-04-26T21:33:28.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.042+0000 7f7c2de4f700 1 -- 172.21.15.46:0/3961942546 wait complete. 2024-04-26T21:33:28.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.043+0000 7f7c2de4f700 1 Processor -- start 2024-04-26T21:33:28.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.043+0000 7f7c2de4f700 1 -- start start 2024-04-26T21:33:28.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.043+0000 7f7c2de4f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c2806b9c0 0x7f7c281023e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:28.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.043+0000 7f7c2de4f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c2806c360 0x7f7c28102920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:28.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.043+0000 7f7c2de4f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c28109490 0x7f7c28102e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:28.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.043+0000 7f7c2de4f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7c28113580 con 0x7f7c2806b9c0 2024-04-26T21:33:28.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.043+0000 7f7c2de4f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7c28113400 con 0x7f7c28109490 2024-04-26T21:33:28.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.043+0000 7f7c2de4f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7c28113700 con 0x7f7c2806c360 2024-04-26T21:33:28.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.043+0000 7f7c277fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c2806b9c0 0x7f7c281023e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:28.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.043+0000 7f7c277fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c2806b9c0 0x7f7c281023e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47582/0 (socket says 172.21.15.46:47582) 2024-04-26T21:33:28.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.043+0000 7f7c277fe700 1 -- 172.21.15.46:0/1978425977 learned_addr learned my addr 172.21.15.46:0/1978425977 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:33:28.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.043+0000 7f7c26ffd700 1 --2- 172.21.15.46:0/1978425977 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c2806c360 0x7f7c28102920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.043+0000 7f7c27fff700 1 --2- 172.21.15.46:0/1978425977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c28109490 0x7f7c28102e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.044+0000 7f7c277fe700 1 -- 172.21.15.46:0/1978425977 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c2806c360 msgr2=0x7f7c28102920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.044+0000 7f7c277fe700 1 --2- 172.21.15.46:0/1978425977 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7c2806c360 0x7f7c28102920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.044+0000 7f7c277fe700 1 -- 172.21.15.46:0/1978425977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c28109490 msgr2=0x7f7c28102e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.044+0000 7f7c277fe700 1 --2- 172.21.15.46:0/1978425977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c28109490 0x7f7c28102e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.044+0000 7f7c277fe700 1 -- 172.21.15.46:0/1978425977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c281a7f90 con 0x7f7c2806b9c0 2024-04-26T21:33:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.044+0000 7f7c277fe700 1 --2- 172.21.15.46:0/1978425977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c2806b9c0 0x7f7c281023e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7c180057c0 tx=0x7f7c18009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.044+0000 7f7c24ff9700 1 -- 172.21.15.46:0/1978425977 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c18026070 con 0x7f7c2806b9c0 2024-04-26T21:33:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.044+0000 7f7c24ff9700 1 -- 172.21.15.46:0/1978425977 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c18021410 con 0x7f7c2806b9c0 2024-04-26T21:33:28.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.044+0000 7f7c2de4f700 1 -- 172.21.15.46:0/1978425977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c281a8220 con 0x7f7c2806b9c0 2024-04-26T21:33:28.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.044+0000 7f7c24ff9700 1 -- 172.21.15.46:0/1978425977 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c18017650 con 0x7f7c2806b9c0 2024-04-26T21:33:28.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.044+0000 7f7c2de4f700 1 -- 172.21.15.46:0/1978425977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c281a8700 con 0x7f7c2806b9c0 2024-04-26T21:33:28.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.045+0000 7f7c2de4f700 1 -- 172.21.15.46:0/1978425977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7c28063b30 con 0x7f7c2806b9c0 2024-04-26T21:33:28.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.046+0000 7f7c24ff9700 1 -- 172.21.15.46:0/1978425977 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7c18029070 con 0x7f7c2806b9c0 2024-04-26T21:33:28.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.047+0000 7f7c24ff9700 1 --2- 172.21.15.46:0/1978425977 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7c100a3cd0 0x7f7c100a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:28.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.047+0000 7f7c26ffd700 1 --2- 172.21.15.46:0/1978425977 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7c100a3cd0 0x7f7c100a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:28.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.047+0000 7f7c24ff9700 1 -- 172.21.15.46:0/1978425977 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f7c180d1550 con 0x7f7c2806b9c0 2024-04-26T21:33:28.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.047+0000 7f7c24ff9700 1 --2- 172.21.15.46:0/1978425977 >> [v2:172.21.15.46:6810/2435181760,v1:172.21.15.46:6811/2435181760] conn(0x7f7c100ad4b0 0x7f7c100af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.047+0000 7f7c24ff9700 1 -- 172.21.15.46:0/1978425977 --> [v2:172.21.15.46:6810/2435181760,v1:172.21.15.46:6811/2435181760] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7c100aff80 con 0x7f7c100ad4b0 2024-04-26T21:33:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.047+0000 7f7c24ff9700 1 -- 172.21.15.46:0/1978425977 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f7c180d1900 con 0x7f7c2806b9c0 2024-04-26T21:33:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.047+0000 7f7c27fff700 1 --2- 172.21.15.46:0/1978425977 >> [v2:172.21.15.46:6810/2435181760,v1:172.21.15.46:6811/2435181760] conn(0x7f7c100ad4b0 0x7f7c100af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.047+0000 7f7c26ffd700 1 --2- 172.21.15.46:0/1978425977 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7c100a3cd0 0x7f7c100a6190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7c28079900 tx=0x7f7c1c00a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.048+0000 7f7c27fff700 1 --2- 172.21.15.46:0/1978425977 >> [v2:172.21.15.46:6810/2435181760,v1:172.21.15.46:6811/2435181760] conn(0x7f7c100ad4b0 0x7f7c100af8d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.048+0000 7f7c24ff9700 1 -- 172.21.15.46:0/1978425977 <== osd.3 v2:172.21.15.46:6810/2435181760 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7c100aff80 con 0x7f7c100ad4b0 2024-04-26T21:33:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.066+0000 7f7c2de4f700 1 -- 172.21.15.46:0/1978425977 --> [v2:172.21.15.46:6810/2435181760,v1:172.21.15.46:6811/2435181760] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7c28063f30 con 0x7f7c100ad4b0 2024-04-26T21:33:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.066+0000 7f7c24ff9700 1 -- 172.21.15.46:0/1978425977 <== osd.3 v2:172.21.15.46:6810/2435181760 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7c28063f30 con 0x7f7c100ad4b0 2024-04-26T21:33:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.066+0000 7f7c2de4f700 1 -- 172.21.15.46:0/1978425977 >> [v2:172.21.15.46:6810/2435181760,v1:172.21.15.46:6811/2435181760] conn(0x7f7c100ad4b0 msgr2=0x7f7c100af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.066+0000 7f7c2de4f700 1 --2- 172.21.15.46:0/1978425977 >> [v2:172.21.15.46:6810/2435181760,v1:172.21.15.46:6811/2435181760] conn(0x7f7c100ad4b0 0x7f7c100af8d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.066+0000 7f7c2de4f700 1 -- 172.21.15.46:0/1978425977 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7c100a3cd0 msgr2=0x7f7c100a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.066+0000 7f7c2de4f700 1 --2- 172.21.15.46:0/1978425977 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7c100a3cd0 0x7f7c100a6190 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7c28079900 tx=0x7f7c1c00a380 comp rx=0 tx=0).stop 2024-04-26T21:33:28.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.066+0000 7f7c2de4f700 1 -- 172.21.15.46:0/1978425977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c2806b9c0 msgr2=0x7f7c281023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:28.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.066+0000 7f7c2de4f700 1 --2- 172.21.15.46:0/1978425977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c2806b9c0 0x7f7c281023e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7c180057c0 tx=0x7f7c18009580 comp rx=0 tx=0).stop 2024-04-26T21:33:28.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.066+0000 7f7c27fff700 1 -- 172.21.15.46:0/1978425977 reap_dead start 2024-04-26T21:33:28.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.067+0000 7f7c2de4f700 1 -- 172.21.15.46:0/1978425977 shutdown_connections 2024-04-26T21:33:28.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.067+0000 7f7c2de4f700 1 -- 172.21.15.46:0/1978425977 >> 172.21.15.46:0/1978425977 conn(0x7f7c28079290 msgr2=0x7f7c28100400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:28.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.067+0000 7f7c2de4f700 1 -- 172.21.15.46:0/1978425977 shutdown_connections 2024-04-26T21:33:28.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:26.067+0000 7f7c2de4f700 1 -- 172.21.15.46:0/1978425977 wait complete. 2024-04-26T21:33:29.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:29 smithi142 ceph-mon[32944]: pgmap v314: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:29.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:29 smithi046 ceph-mon[29201]: pgmap v314: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:29.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:29 smithi050 ceph-mon[32753]: pgmap v314: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:29.526 INFO:teuthology.orchestra.run.smithi046.stdout:107374182463 2024-04-26T21:33:29.527 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-26T21:33:29.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.406+0000 7fb3a125f700 1 -- 172.21.15.46:0/2547694738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb39c06baf0 msgr2=0x7fb39c06bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:29.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.406+0000 7fb3a125f700 1 --2- 172.21.15.46:0/2547694738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb39c06baf0 0x7fb39c06bf50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb390009990 tx=0x7fb390009ca0 comp rx=0 tx=0).stop 2024-04-26T21:33:29.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.406+0000 7fb3a125f700 1 -- 172.21.15.46:0/2547694738 shutdown_connections 2024-04-26T21:33:29.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.406+0000 7fb3a125f700 1 --2- 172.21.15.46:0/2547694738 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb39c06c680 0x7fb39c105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:29.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.406+0000 7fb3a125f700 1 --2- 172.21.15.46:0/2547694738 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb39c108370 0x7fb39c108750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:29.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.406+0000 7fb3a125f700 1 --2- 172.21.15.46:0/2547694738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb39c06baf0 0x7fb39c06bf50 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:29.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.406+0000 7fb3a125f700 1 -- 172.21.15.46:0/2547694738 >> 172.21.15.46:0/2547694738 conn(0x7fb39c0792d0 msgr2=0x7fb39c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:29.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.406+0000 7fb3a125f700 1 -- 172.21.15.46:0/2547694738 shutdown_connections 2024-04-26T21:33:29.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.406+0000 7fb3a125f700 1 -- 172.21.15.46:0/2547694738 wait complete. 2024-04-26T21:33:29.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.407+0000 7fb3a125f700 1 Processor -- start 2024-04-26T21:33:29.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.407+0000 7fb3a125f700 1 -- start start 2024-04-26T21:33:29.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.407+0000 7fb3a125f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb39c06baf0 0x7fb39c1a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:29.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.407+0000 7fb3a125f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb39c06c680 0x7fb39c1a0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:29.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.407+0000 7fb3a125f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb39c108370 0x7fb39c1a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:29.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.407+0000 7fb3a125f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c119c10 con 0x7fb39c06baf0 2024-04-26T21:33:29.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.407+0000 7fb3a125f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c119a90 con 0x7fb39c06c680 2024-04-26T21:33:29.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.407+0000 7fb3a125f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c119d90 con 0x7fb39c108370 2024-04-26T21:33:29.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.408+0000 7fb39ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb39c06baf0 0x7fb39c1a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:29.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.408+0000 7fb39ad9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb39c06baf0 0x7fb39c1a01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47618/0 (socket says 172.21.15.46:47618) 2024-04-26T21:33:29.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.408+0000 7fb39ad9d700 1 -- 172.21.15.46:0/3876267129 learned_addr learned my addr 172.21.15.46:0/3876267129 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:33:29.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.408+0000 7fb39a59c700 1 --2- 172.21.15.46:0/3876267129 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb39c06c680 0x7fb39c1a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:29.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.408+0000 7fb39b59e700 1 --2- 172.21.15.46:0/3876267129 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb39c108370 0x7fb39c1a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:29.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.408+0000 7fb39ad9d700 1 -- 172.21.15.46:0/3876267129 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb39c108370 msgr2=0x7fb39c1a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:29.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.408+0000 7fb39ad9d700 1 --2- 172.21.15.46:0/3876267129 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb39c108370 0x7fb39c1a4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:29.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.408+0000 7fb39ad9d700 1 -- 172.21.15.46:0/3876267129 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb39c06c680 msgr2=0x7fb39c1a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:29.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.408+0000 7fb39ad9d700 1 --2- 172.21.15.46:0/3876267129 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb39c06c680 0x7fb39c1a0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:29.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.408+0000 7fb39ad9d700 1 -- 172.21.15.46:0/3876267129 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb39c1a51f0 con 0x7fb39c06baf0 2024-04-26T21:33:29.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.408+0000 7fb39a59c700 1 --2- 172.21.15.46:0/3876267129 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb39c06c680 0x7fb39c1a0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:33:29.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.408+0000 7fb39ad9d700 1 --2- 172.21.15.46:0/3876267129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb39c06baf0 0x7fb39c1a01e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb38c00d980 tx=0x7fb38c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:29.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.408+0000 7fb383fff700 1 -- 172.21.15.46:0/3876267129 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38c016070 con 0x7fb39c06baf0 2024-04-26T21:33:29.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.408+0000 7fb3a125f700 1 -- 172.21.15.46:0/3876267129 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb39c1a54e0 con 0x7fb39c06baf0 2024-04-26T21:33:29.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.408+0000 7fb383fff700 1 -- 172.21.15.46:0/3876267129 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb38c004830 con 0x7fb39c06baf0 2024-04-26T21:33:29.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.408+0000 7fb383fff700 1 -- 172.21.15.46:0/3876267129 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38c005540 con 0x7fb39c06baf0 2024-04-26T21:33:29.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.408+0000 7fb3a125f700 1 -- 172.21.15.46:0/3876267129 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb39c1acd60 con 0x7fb39c06baf0 2024-04-26T21:33:29.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.409+0000 7fb3a125f700 1 -- 172.21.15.46:0/3876267129 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb39c102050 con 0x7fb39c06baf0 2024-04-26T21:33:29.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.411+0000 7fb383fff700 1 -- 172.21.15.46:0/3876267129 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb38c004220 con 0x7fb39c06baf0 2024-04-26T21:33:29.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.411+0000 7fb383fff700 1 --2- 172.21.15.46:0/3876267129 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb3840a3cd0 0x7fb3840a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:29.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.411+0000 7fb39a59c700 1 --2- 172.21.15.46:0/3876267129 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb3840a3cd0 0x7fb3840a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:29.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.411+0000 7fb383fff700 1 -- 172.21.15.46:0/3876267129 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fb38c0d07c0 con 0x7fb39c06baf0 2024-04-26T21:33:29.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.412+0000 7fb383fff700 1 --2- 172.21.15.46:0/3876267129 >> [v2:172.21.15.142:6808/4113107737,v1:172.21.15.142:6809/4113107737] conn(0x7fb3840ad510 0x7fb3840af930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:29.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.412+0000 7fb383fff700 1 -- 172.21.15.46:0/3876267129 --> [v2:172.21.15.142:6808/4113107737,v1:172.21.15.142:6809/4113107737] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb38c015ec0 con 0x7fb3840ad510 2024-04-26T21:33:29.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.412+0000 7fb383fff700 1 -- 172.21.15.46:0/3876267129 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fb38c0d0ca0 con 0x7fb39c06baf0 2024-04-26T21:33:29.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.412+0000 7fb39a59c700 1 --2- 172.21.15.46:0/3876267129 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb3840a3cd0 0x7fb3840a6190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb3900098e0 tx=0x7fb390005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:29.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.412+0000 7fb39b59e700 1 --2- 172.21.15.46:0/3876267129 >> [v2:172.21.15.142:6808/4113107737,v1:172.21.15.142:6809/4113107737] conn(0x7fb3840ad510 0x7fb3840af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:29.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.412+0000 7fb39b59e700 1 --2- 172.21.15.46:0/3876267129 >> [v2:172.21.15.142:6808/4113107737,v1:172.21.15.142:6809/4113107737] conn(0x7fb3840ad510 0x7fb3840af930 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:29.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.413+0000 7fb383fff700 1 -- 172.21.15.46:0/3876267129 <== osd.5 v2:172.21.15.142:6808/4113107737 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb38c015ec0 con 0x7fb3840ad510 2024-04-26T21:33:29.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.430+0000 7fb3a125f700 1 -- 172.21.15.46:0/3876267129 --> [v2:172.21.15.142:6808/4113107737,v1:172.21.15.142:6809/4113107737] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb39c06b3a0 con 0x7fb3840ad510 2024-04-26T21:33:29.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.431+0000 7fb383fff700 1 -- 172.21.15.46:0/3876267129 <== osd.5 v2:172.21.15.142:6808/4113107737 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb39c06b3a0 con 0x7fb3840ad510 2024-04-26T21:33:29.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.431+0000 7fb3a125f700 1 -- 172.21.15.46:0/3876267129 >> [v2:172.21.15.142:6808/4113107737,v1:172.21.15.142:6809/4113107737] conn(0x7fb3840ad510 msgr2=0x7fb3840af930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:29.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.431+0000 7fb3a125f700 1 --2- 172.21.15.46:0/3876267129 >> [v2:172.21.15.142:6808/4113107737,v1:172.21.15.142:6809/4113107737] conn(0x7fb3840ad510 0x7fb3840af930 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:29.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.431+0000 7fb3a125f700 1 -- 172.21.15.46:0/3876267129 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb3840a3cd0 msgr2=0x7fb3840a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:29.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.431+0000 7fb3a125f700 1 --2- 172.21.15.46:0/3876267129 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb3840a3cd0 0x7fb3840a6190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb3900098e0 tx=0x7fb390005960 comp rx=0 tx=0).stop 2024-04-26T21:33:29.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.431+0000 7fb3a125f700 1 -- 172.21.15.46:0/3876267129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb39c06baf0 msgr2=0x7fb39c1a01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:29.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.431+0000 7fb3a125f700 1 --2- 172.21.15.46:0/3876267129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb39c06baf0 0x7fb39c1a01e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb38c00d980 tx=0x7fb38c00dc90 comp rx=0 tx=0).stop 2024-04-26T21:33:29.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.431+0000 7fb39b59e700 1 -- 172.21.15.46:0/3876267129 reap_dead start 2024-04-26T21:33:29.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.431+0000 7fb3a125f700 1 -- 172.21.15.46:0/3876267129 shutdown_connections 2024-04-26T21:33:29.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.431+0000 7fb3a125f700 1 -- 172.21.15.46:0/3876267129 >> 172.21.15.46:0/3876267129 conn(0x7fb39c0792d0 msgr2=0x7fb39c1016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:29.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.431+0000 7fb3a125f700 1 -- 172.21.15.46:0/3876267129 shutdown_connections 2024-04-26T21:33:29.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:28.431+0000 7fb3a125f700 1 -- 172.21.15.46:0/3876267129 wait complete. 2024-04-26T21:33:30.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:30 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-26T21:33:30.322 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:30 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-26T21:33:30.322 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:30 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-04-26T21:33:30.322 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:30 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-26T21:33:30.322 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:30 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8, 5, 9]}]: dispatch 2024-04-26T21:33:30.322 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:30 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-04-26T21:33:30.322 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:30 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2024-04-26T21:33:30.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:30 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-26T21:33:30.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:30 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-26T21:33:30.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:30 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-04-26T21:33:30.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:30 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-26T21:33:30.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:30 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8, 5, 9]}]: dispatch 2024-04-26T21:33:30.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:30 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-04-26T21:33:30.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:30 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2024-04-26T21:33:30.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:30 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-26T21:33:30.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:30 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-26T21:33:30.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:30 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-04-26T21:33:30.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:30 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-26T21:33:30.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:30 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8, 5, 9]}]: dispatch 2024-04-26T21:33:30.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:30 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-04-26T21:33:30.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:30 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]: dispatch 2024-04-26T21:33:30.781 INFO:teuthology.orchestra.run.smithi046.stdout:34359738445 2024-04-26T21:33:30.782 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-26T21:33:31.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:31 smithi046 ceph-mon[29201]: pgmap v315: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:33:31.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-26T21:33:31.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-26T21:33:31.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-04-26T21:33:31.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-26T21:33:31.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8, 5, 9]}]': finished 2024-04-26T21:33:31.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-04-26T21:33:31.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:31 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-04-26T21:33:31.304 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:31 smithi046 ceph-mon[29201]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T21:33:31.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.311+0000 7fdf65f10700 1 -- 172.21.15.46:0/3062526221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf601025a0 msgr2=0x7fdf60102980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:31.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.311+0000 7fdf65f10700 1 --2- 172.21.15.46:0/3062526221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf601025a0 0x7fdf60102980 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdf500099c0 tx=0x7fdf50009cd0 comp rx=0 tx=0).stop 2024-04-26T21:33:31.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.311+0000 7fdf65f10700 1 -- 172.21.15.46:0/3062526221 shutdown_connections 2024-04-26T21:33:31.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.311+0000 7fdf65f10700 1 --2- 172.21.15.46:0/3062526221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf601025a0 0x7fdf60102980 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:31.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.311+0000 7fdf65f10700 1 --2- 172.21.15.46:0/3062526221 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf60107680 0x7fdf6010f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:31.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.311+0000 7fdf65f10700 1 --2- 172.21.15.46:0/3062526221 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf60102f50 0x7fdf60106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:31.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.311+0000 7fdf65f10700 1 -- 172.21.15.46:0/3062526221 >> 172.21.15.46:0/3062526221 conn(0x7fdf600793c0 msgr2=0x7fdf600797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:31.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.311+0000 7fdf65f10700 1 -- 172.21.15.46:0/3062526221 shutdown_connections 2024-04-26T21:33:31.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.311+0000 7fdf65f10700 1 -- 172.21.15.46:0/3062526221 wait complete. 2024-04-26T21:33:31.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.312+0000 7fdf65f10700 1 Processor -- start 2024-04-26T21:33:31.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.312+0000 7fdf65f10700 1 -- start start 2024-04-26T21:33:31.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.312+0000 7fdf65f10700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf601025a0 0x7fdf60104040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:31.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.312+0000 7fdf65f10700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf60102f50 0x7fdf60104580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:31.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.312+0000 7fdf65f10700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf60107680 0x7fdf60104ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:31.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.312+0000 7fdf65f10700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fdf60112540 con 0x7fdf60102f50 2024-04-26T21:33:31.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.312+0000 7fdf65f10700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdf601123c0 con 0x7fdf60107680 2024-04-26T21:33:31.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.312+0000 7fdf65f10700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fdf601126c0 con 0x7fdf601025a0 2024-04-26T21:33:31.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.312+0000 7fdf5effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf60102f50 0x7fdf60104580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:31.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.312+0000 7fdf5effd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf60102f50 0x7fdf60104580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47648/0 (socket says 172.21.15.46:47648) 2024-04-26T21:33:31.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.312+0000 7fdf5effd700 1 -- 172.21.15.46:0/1465929392 learned_addr learned my addr 172.21.15.46:0/1465929392 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:33:31.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.312+0000 7fdf5ffff700 1 --2- 172.21.15.46:0/1465929392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf60107680 0x7fdf60104ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:31.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.312+0000 7fdf5f7fe700 1 --2- 172.21.15.46:0/1465929392 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf601025a0 0x7fdf60104040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:31.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.312+0000 7fdf5effd700 1 -- 172.21.15.46:0/1465929392 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf601025a0 msgr2=0x7fdf60104040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:31.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.312+0000 7fdf5effd700 1 --2- 172.21.15.46:0/1465929392 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf601025a0 0x7fdf60104040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:31.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.313+0000 7fdf5effd700 1 -- 172.21.15.46:0/1465929392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf60107680 msgr2=0x7fdf60104ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:31.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.313+0000 7fdf5effd700 1 --2- 172.21.15.46:0/1465929392 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf60107680 0x7fdf60104ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:31.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.313+0000 7fdf5effd700 1 -- 172.21.15.46:0/1465929392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf601a8080 con 0x7fdf60102f50 2024-04-26T21:33:31.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.313+0000 7fdf5f7fe700 1 --2- 172.21.15.46:0/1465929392 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fdf601025a0 0x7fdf60104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:33:31.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.313+0000 7fdf5effd700 1 --2- 172.21.15.46:0/1465929392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf60102f50 0x7fdf60104580 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fdf5400d980 tx=0x7fdf5400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:31.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.313+0000 7fdf5cff9700 1 -- 172.21.15.46:0/1465929392 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf54017070 con 0x7fdf60102f50 2024-04-26T21:33:31.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.313+0000 7fdf5cff9700 1 -- 172.21.15.46:0/1465929392 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf54009de0 con 0x7fdf60102f50 2024-04-26T21:33:31.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.313+0000 7fdf65f10700 1 -- 172.21.15.46:0/1465929392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf601a8370 con 0x7fdf60102f50 2024-04-26T21:33:31.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.313+0000 7fdf5cff9700 1 -- 172.21.15.46:0/1465929392 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5400f3c0 con 0x7fdf60102f50 2024-04-26T21:33:31.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.313+0000 7fdf65f10700 1 -- 172.21.15.46:0/1465929392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf601a88b0 con 0x7fdf60102f50 2024-04-26T21:33:31.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.314+0000 7fdf65f10700 1 -- 172.21.15.46:0/1465929392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdf4c001030 con 0x7fdf60102f50 2024-04-26T21:33:31.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.315+0000 7fdf5cff9700 1 -- 172.21.15.46:0/1465929392 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fdf54009930 con 0x7fdf60102f50 2024-04-26T21:33:31.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.316+0000 7fdf5cff9700 1 --2- 172.21.15.46:0/1465929392 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fdf480a3cd0 0x7fdf480a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:31.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.316+0000 7fdf5f7fe700 1 --2- 172.21.15.46:0/1465929392 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fdf480a3cd0 0x7fdf480a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:31.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.316+0000 7fdf5cff9700 1 -- 172.21.15.46:0/1465929392 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7fdf540d03d0 con 0x7fdf60102f50 2024-04-26T21:33:31.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.317+0000 7fdf5cff9700 1 --2- 172.21.15.46:0/1465929392 >> [v2:172.21.15.142:6800/1513279137,v1:172.21.15.142:6801/1513279137] conn(0x7fdf480ad790 0x7fdf480afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:31.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.317+0000 7fdf5cff9700 1 -- 172.21.15.46:0/1465929392 --> [v2:172.21.15.142:6800/1513279137,v1:172.21.15.142:6801/1513279137] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdf480b0300 con 0x7fdf480ad790 2024-04-26T21:33:31.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.317+0000 7fdf5cff9700 1 -- 172.21.15.46:0/1465929392 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7fdf540d0780 con 0x7fdf60102f50 2024-04-26T21:33:31.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.317+0000 7fdf5f7fe700 1 --2- 172.21.15.46:0/1465929392 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fdf480a3cd0 0x7fdf480a6190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fdf50003800 tx=0x7fdf50005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:31.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.317+0000 7fdf5ffff700 1 --2- 172.21.15.46:0/1465929392 >> [v2:172.21.15.142:6800/1513279137,v1:172.21.15.142:6801/1513279137] conn(0x7fdf480ad790 0x7fdf480afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:31.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.318+0000 7fdf5ffff700 1 --2- 172.21.15.46:0/1465929392 >> [v2:172.21.15.142:6800/1513279137,v1:172.21.15.142:6801/1513279137] conn(0x7fdf480ad790 0x7fdf480afc30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:31.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.318+0000 7fdf5cff9700 1 -- 172.21.15.46:0/1465929392 <== osd.2 v2:172.21.15.142:6800/1513279137 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdf480b0300 con 0x7fdf480ad790 2024-04-26T21:33:31.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.338+0000 7fdf65f10700 1 -- 172.21.15.46:0/1465929392 --> [v2:172.21.15.142:6800/1513279137,v1:172.21.15.142:6801/1513279137] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdf4c002e60 con 0x7fdf480ad790 2024-04-26T21:33:31.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.339+0000 7fdf5cff9700 1 -- 172.21.15.46:0/1465929392 <== osd.2 v2:172.21.15.142:6800/1513279137 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdf4c002e60 con 0x7fdf480ad790 2024-04-26T21:33:31.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.339+0000 7fdf65f10700 1 -- 172.21.15.46:0/1465929392 >> [v2:172.21.15.142:6800/1513279137,v1:172.21.15.142:6801/1513279137] conn(0x7fdf480ad790 msgr2=0x7fdf480afc30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:31.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.339+0000 7fdf65f10700 1 --2- 172.21.15.46:0/1465929392 >> [v2:172.21.15.142:6800/1513279137,v1:172.21.15.142:6801/1513279137] conn(0x7fdf480ad790 0x7fdf480afc30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:31.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.339+0000 7fdf65f10700 1 -- 172.21.15.46:0/1465929392 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fdf480a3cd0 msgr2=0x7fdf480a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:31.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.339+0000 7fdf65f10700 1 --2- 172.21.15.46:0/1465929392 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fdf480a3cd0 0x7fdf480a6190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fdf50003800 tx=0x7fdf50005fd0 comp rx=0 tx=0).stop 2024-04-26T21:33:31.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.339+0000 7fdf65f10700 1 -- 172.21.15.46:0/1465929392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf60102f50 msgr2=0x7fdf60104580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:31.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.339+0000 7fdf65f10700 1 --2- 172.21.15.46:0/1465929392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf60102f50 0x7fdf60104580 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fdf5400d980 tx=0x7fdf5400dc90 comp rx=0 tx=0).stop 2024-04-26T21:33:31.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.339+0000 7fdf5ffff700 1 -- 172.21.15.46:0/1465929392 reap_dead start 2024-04-26T21:33:31.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.339+0000 7fdf65f10700 1 -- 172.21.15.46:0/1465929392 shutdown_connections 2024-04-26T21:33:31.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.339+0000 7fdf65f10700 1 -- 172.21.15.46:0/1465929392 >> 172.21.15.46:0/1465929392 conn(0x7fdf600793c0 msgr2=0x7fdf6006bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:31.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.339+0000 7fdf65f10700 1 -- 172.21.15.46:0/1465929392 shutdown_connections 2024-04-26T21:33:31.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.339+0000 7fdf65f10700 1 -- 172.21.15.46:0/1465929392 wait complete. 2024-04-26T21:33:31.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:31 smithi050 ceph-mon[32753]: pgmap v315: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:33:31.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-26T21:33:31.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-26T21:33:31.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-04-26T21:33:31.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-26T21:33:31.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8, 5, 9]}]': finished 2024-04-26T21:33:31.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-04-26T21:33:31.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:31 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-04-26T21:33:31.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:31 smithi050 ceph-mon[32753]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T21:33:31.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.207+0000 7fccb7e19700 1 -- 172.21.15.46:0/112876413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccb0108380 msgr2=0x7fccb0108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:31.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.207+0000 7fccb7e19700 1 --2- 172.21.15.46:0/112876413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccb0108380 0x7fccb0108760 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fccac009990 tx=0x7fccac009ca0 comp rx=0 tx=0).stop 2024-04-26T21:33:31.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.207+0000 7fccb7e19700 1 -- 172.21.15.46:0/112876413 shutdown_connections 2024-04-26T21:33:31.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.207+0000 7fccb7e19700 1 --2- 172.21.15.46:0/112876413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccb0108380 0x7fccb0108760 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:31.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.207+0000 7fccb7e19700 1 --2- 172.21.15.46:0/112876413 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fccb0102cd0 0x7fccb010f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:31.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.207+0000 7fccb7e19700 1 --2- 172.21.15.46:0/112876413 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fccb0102310 0x7fccb0102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:31.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.207+0000 7fccb7e19700 1 -- 172.21.15.46:0/112876413 >> 172.21.15.46:0/112876413 conn(0x7fccb0079290 msgr2=0x7fccb00796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:31.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.208+0000 7fccb7e19700 1 -- 172.21.15.46:0/112876413 shutdown_connections 2024-04-26T21:33:31.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.208+0000 7fccb7e19700 1 -- 172.21.15.46:0/112876413 wait complete. 2024-04-26T21:33:31.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.208+0000 7fccb7e19700 1 Processor -- start 2024-04-26T21:33:31.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.208+0000 7fccb7e19700 1 -- start start 2024-04-26T21:33:31.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.209+0000 7fccb7e19700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fccb0102310 0x7fccb019a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:31.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.209+0000 7fccb7e19700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccb0102cd0 0x7fccb019ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:31.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.209+0000 7fccb7e19700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fccb0108380 0x7fccb01a5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:31.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.209+0000 7fccb7e19700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fccb01124a0 con 0x7fccb0102cd0 2024-04-26T21:33:31.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.209+0000 7fccb7e19700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fccb0112320 con 0x7fccb0102310 2024-04-26T21:33:31.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.209+0000 7fccb7e19700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fccb0112620 con 0x7fccb0108380 2024-04-26T21:33:31.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.209+0000 7fccb53b4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccb0102cd0 0x7fccb019ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:31.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.209+0000 7fccb53b4700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccb0102cd0 0x7fccb019ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47632/0 (socket says 172.21.15.46:47632) 2024-04-26T21:33:31.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.209+0000 7fccb53b4700 1 -- 172.21.15.46:0/1363776733 learned_addr learned my addr 172.21.15.46:0/1363776733 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:33:31.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.209+0000 7fccb5bb5700 1 --2- 172.21.15.46:0/1363776733 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fccb0102310 0x7fccb019a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:31.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.209+0000 7fccb63b6700 1 --2- 172.21.15.46:0/1363776733 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fccb0108380 0x7fccb01a5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:31.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.209+0000 7fccb53b4700 1 -- 172.21.15.46:0/1363776733 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fccb0108380 msgr2=0x7fccb01a5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:31.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.209+0000 7fccb53b4700 1 --2- 172.21.15.46:0/1363776733 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fccb0108380 0x7fccb01a5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:31.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.209+0000 7fccb53b4700 1 -- 172.21.15.46:0/1363776733 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fccb0102310 msgr2=0x7fccb019a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:31.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.209+0000 7fccb53b4700 1 --2- 172.21.15.46:0/1363776733 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fccb0102310 0x7fccb019a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:31.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.209+0000 7fccb53b4700 1 -- 172.21.15.46:0/1363776733 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccb01a5ec0 con 0x7fccb0102cd0 2024-04-26T21:33:31.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.209+0000 7fccb63b6700 1 --2- 172.21.15.46:0/1363776733 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fccb0108380 0x7fccb01a5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:33:31.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.209+0000 7fccb53b4700 1 --2- 172.21.15.46:0/1363776733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccb0102cd0 0x7fccb019ae40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcca000d980 tx=0x7fcca000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:31.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.210+0000 7fcca6ffd700 1 -- 172.21.15.46:0/1363776733 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca0016070 con 0x7fccb0102cd0 2024-04-26T21:33:31.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.210+0000 7fccb7e19700 1 -- 172.21.15.46:0/1363776733 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccb01a60f0 con 0x7fccb0102cd0 2024-04-26T21:33:31.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.210+0000 7fcca6ffd700 1 -- 172.21.15.46:0/1363776733 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcca0004830 con 0x7fccb0102cd0 2024-04-26T21:33:31.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.210+0000 7fcca6ffd700 1 -- 172.21.15.46:0/1363776733 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca00054c0 con 0x7fccb0102cd0 2024-04-26T21:33:31.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.210+0000 7fccb7e19700 1 -- 172.21.15.46:0/1363776733 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccb01a6680 con 0x7fccb0102cd0 2024-04-26T21:33:31.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.210+0000 7fccb7e19700 1 -- 172.21.15.46:0/1363776733 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fccb0063b30 con 0x7fccb0102cd0 2024-04-26T21:33:31.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.212+0000 7fcca6ffd700 1 -- 172.21.15.46:0/1363776733 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcca0004220 con 0x7fccb0102cd0 2024-04-26T21:33:31.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.212+0000 7fcca6ffd700 1 --2- 172.21.15.46:0/1363776733 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcc9c0a3d50 0x7fcc9c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:31.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.213+0000 7fcca6ffd700 1 -- 172.21.15.46:0/1363776733 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7fcca00d0970 con 0x7fccb0102cd0 2024-04-26T21:33:31.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.213+0000 7fccb5bb5700 1 --2- 172.21.15.46:0/1363776733 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcc9c0a3d50 0x7fcc9c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:31.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.213+0000 7fcca6ffd700 1 --2- 172.21.15.46:0/1363776733 >> [v2:172.21.15.50:6824/3135765006,v1:172.21.15.50:6825/3135765006] conn(0x7fcc9c0ad810 0x7fcc9c0afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:31.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.213+0000 7fcca6ffd700 1 -- 172.21.15.46:0/1363776733 --> [v2:172.21.15.50:6824/3135765006,v1:172.21.15.50:6825/3135765006] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcca0013ec0 con 0x7fcc9c0ad810 2024-04-26T21:33:31.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.213+0000 7fcca6ffd700 1 -- 172.21.15.46:0/1363776733 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7fcca00d0dc0 con 0x7fccb0102cd0 2024-04-26T21:33:31.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.213+0000 7fccb5bb5700 1 --2- 172.21.15.46:0/1363776733 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcc9c0a3d50 0x7fcc9c0a6210 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fccac0098e0 tx=0x7fccac005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:31.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.213+0000 7fccb63b6700 1 --2- 172.21.15.46:0/1363776733 >> [v2:172.21.15.50:6824/3135765006,v1:172.21.15.50:6825/3135765006] conn(0x7fcc9c0ad810 0x7fcc9c0afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:31.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.214+0000 7fccb63b6700 1 --2- 172.21.15.46:0/1363776733 >> [v2:172.21.15.50:6824/3135765006,v1:172.21.15.50:6825/3135765006] conn(0x7fcc9c0ad810 0x7fcc9c0afcb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:31.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.215+0000 7fcca6ffd700 1 -- 172.21.15.46:0/1363776733 <== osd.10 v2:172.21.15.50:6824/3135765006 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcca0013ec0 con 0x7fcc9c0ad810 2024-04-26T21:33:31.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.231+0000 7fccb7e19700 1 -- 172.21.15.46:0/1363776733 --> [v2:172.21.15.50:6824/3135765006,v1:172.21.15.50:6825/3135765006] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fccb0063f30 con 0x7fcc9c0ad810 2024-04-26T21:33:31.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.232+0000 7fcca6ffd700 1 -- 172.21.15.46:0/1363776733 <== osd.10 v2:172.21.15.50:6824/3135765006 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fccb0063f30 con 0x7fcc9c0ad810 2024-04-26T21:33:31.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.232+0000 7fccb7e19700 1 -- 172.21.15.46:0/1363776733 >> [v2:172.21.15.50:6824/3135765006,v1:172.21.15.50:6825/3135765006] conn(0x7fcc9c0ad810 msgr2=0x7fcc9c0afcb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:31.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.232+0000 7fccb7e19700 1 --2- 172.21.15.46:0/1363776733 >> [v2:172.21.15.50:6824/3135765006,v1:172.21.15.50:6825/3135765006] conn(0x7fcc9c0ad810 0x7fcc9c0afcb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:31.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.232+0000 7fccb7e19700 1 -- 172.21.15.46:0/1363776733 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcc9c0a3d50 msgr2=0x7fcc9c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:31.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.232+0000 7fccb7e19700 1 --2- 172.21.15.46:0/1363776733 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcc9c0a3d50 0x7fcc9c0a6210 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fccac0098e0 tx=0x7fccac005960 comp rx=0 tx=0).stop 2024-04-26T21:33:31.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.232+0000 7fccb7e19700 1 -- 172.21.15.46:0/1363776733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccb0102cd0 msgr2=0x7fccb019ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:31.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.232+0000 7fccb7e19700 1 --2- 172.21.15.46:0/1363776733 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccb0102cd0 0x7fccb019ae40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcca000d980 tx=0x7fcca000dc90 comp rx=0 tx=0).stop 2024-04-26T21:33:31.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.232+0000 7fccb63b6700 1 -- 172.21.15.46:0/1363776733 reap_dead start 2024-04-26T21:33:31.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.232+0000 7fccb7e19700 1 -- 172.21.15.46:0/1363776733 shutdown_connections 2024-04-26T21:33:31.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.232+0000 7fccb7e19700 1 -- 172.21.15.46:0/1363776733 >> 172.21.15.46:0/1363776733 conn(0x7fccb0079290 msgr2=0x7fccb0101070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:31.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.232+0000 7fccb7e19700 1 -- 172.21.15.46:0/1363776733 shutdown_connections 2024-04-26T21:33:31.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:31.232+0000 7fccb7e19700 1 -- 172.21.15.46:0/1363776733 wait complete. 2024-04-26T21:33:31.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:31 smithi142 ceph-mon[32944]: pgmap v315: 97 pgs: 97 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:33:31.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-26T21:33:31.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-26T21:33:31.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-04-26T21:33:31.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-26T21:33:31.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8, 5, 9]}]': finished 2024-04-26T21:33:31.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-04-26T21:33:31.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:31 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [3, 9]}]': finished 2024-04-26T21:33:31.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:31 smithi142 ceph-mon[32944]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T21:33:32.058 INFO:teuthology.orchestra.run.smithi046.stdout:309237645340 2024-04-26T21:33:32.058 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-26T21:33:32.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:32 smithi046 ceph-mon[29201]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T21:33:32.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:32 smithi050 ceph-mon[32753]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T21:33:32.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:32 smithi142 ceph-mon[32944]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T21:33:32.686 INFO:teuthology.orchestra.run.smithi046.stdout:55834574922 2024-04-26T21:33:32.686 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-26T21:33:33.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:33 smithi050 ceph-mon[32753]: pgmap v318: 97 pgs: 4 peering, 93 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-04-26T21:33:33.461 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:33 smithi046 ceph-mon[29201]: pgmap v318: 97 pgs: 4 peering, 93 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-04-26T21:33:33.497 INFO:teuthology.orchestra.run.smithi046.stdout:197568495664 2024-04-26T21:33:33.497 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-26T21:33:33.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:33 smithi142 ceph-mon[32944]: pgmap v318: 97 pgs: 4 peering, 93 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-04-26T21:33:33.962 INFO:teuthology.orchestra.run.smithi046.stdout:77309411396 2024-04-26T21:33:33.962 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-26T21:33:34.404 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:33:34.405 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:33:34.408 INFO:teuthology.orchestra.run.smithi046.stdout:163208757302 2024-04-26T21:33:34.408 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:33:34.409 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-26T21:33:34.409 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:33:34.415 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:33:34.486 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:33:35.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:35 smithi046 ceph-mon[29201]: pgmap v319: 97 pgs: 4 peering, 93 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:35.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:35 smithi050 ceph-mon[32753]: pgmap v319: 97 pgs: 4 peering, 93 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:35.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:35 smithi142 ceph-mon[32944]: pgmap v319: 97 pgs: 4 peering, 93 active+clean; 629 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-04-26T21:33:37.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:37 smithi046 ceph-mon[29201]: pgmap v320: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s; 13 B/s, 1 objects/s recovering 2024-04-26T21:33:37.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:37 smithi050 ceph-mon[32753]: pgmap v320: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s; 13 B/s, 1 objects/s recovering 2024-04-26T21:33:37.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:37 smithi142 ceph-mon[32944]: pgmap v320: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s; 13 B/s, 1 objects/s recovering 2024-04-26T21:33:39.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:39 smithi050 ceph-mon[32753]: pgmap v321: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s; 13 B/s, 1 objects/s recovering 2024-04-26T21:33:39.541 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:39 smithi046 ceph-mon[29201]: pgmap v321: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s; 13 B/s, 1 objects/s recovering 2024-04-26T21:33:39.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:39 smithi142 ceph-mon[32944]: pgmap v321: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s; 13 B/s, 1 objects/s recovering 2024-04-26T21:33:41.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.774+0000 7f8ba7ea2700 1 -- 172.21.15.46:0/252396389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ba0102320 msgr2=0x7f8ba01027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.774+0000 7f8ba7ea2700 1 --2- 172.21.15.46:0/252396389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ba0102320 0x7f8ba01027a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8b90009990 tx=0x7f8b90009ca0 comp rx=0 tx=0).stop 2024-04-26T21:33:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.774+0000 7f8ba7ea2700 1 -- 172.21.15.46:0/252396389 shutdown_connections 2024-04-26T21:33:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.774+0000 7f8ba7ea2700 1 --2- 172.21.15.46:0/252396389 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ba0102ce0 0x7f8ba010f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.774+0000 7f8ba7ea2700 1 --2- 172.21.15.46:0/252396389 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8ba0108390 0x7f8ba0108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.774+0000 7f8ba7ea2700 1 --2- 172.21.15.46:0/252396389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ba0102320 0x7f8ba01027a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.774+0000 7f8ba7ea2700 1 -- 172.21.15.46:0/252396389 >> 172.21.15.46:0/252396389 conn(0x7f8ba00792d0 msgr2=0x7f8ba00796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.774+0000 7f8ba7ea2700 1 -- 172.21.15.46:0/252396389 shutdown_connections 2024-04-26T21:33:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.774+0000 7f8ba7ea2700 1 -- 172.21.15.46:0/252396389 wait complete. 2024-04-26T21:33:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.775+0000 7f8ba7ea2700 1 Processor -- start 2024-04-26T21:33:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.775+0000 7f8ba7ea2700 1 -- start start 2024-04-26T21:33:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.776+0000 7f8ba7ea2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ba0102320 0x7f8ba019a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:41.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.776+0000 7f8ba7ea2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8ba0102ce0 0x7f8ba019aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:41.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.776+0000 7f8ba7ea2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ba0108390 0x7f8ba01a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:41.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.776+0000 7f8ba7ea2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba0112530 con 0x7f8ba0102320 2024-04-26T21:33:41.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.776+0000 7f8ba7ea2700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba01123b0 con 0x7f8ba0108390 2024-04-26T21:33:41.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.776+0000 7f8ba7ea2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba01126b0 con 0x7f8ba0102ce0 2024-04-26T21:33:41.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.776+0000 7f8ba5c3e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ba0102320 0x7f8ba019a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:41.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.776+0000 7f8ba5c3e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ba0102320 0x7f8ba019a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43604/0 (socket says 172.21.15.46:43604) 2024-04-26T21:33:41.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.776+0000 7f8ba5c3e700 1 -- 172.21.15.46:0/789350519 learned_addr learned my addr 172.21.15.46:0/789350519 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:33:41.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.776+0000 7f8ba543d700 1 --2- 172.21.15.46:0/789350519 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8ba0102ce0 0x7f8ba019aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:41.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.776+0000 7f8ba643f700 1 --2- 172.21.15.46:0/789350519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ba0108390 0x7f8ba01a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:41.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.776+0000 7f8ba5c3e700 1 -- 172.21.15.46:0/789350519 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8ba0102ce0 msgr2=0x7f8ba019aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:41.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.776+0000 7f8ba5c3e700 1 --2- 172.21.15.46:0/789350519 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8ba0102ce0 0x7f8ba019aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:41.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.776+0000 7f8ba5c3e700 1 -- 172.21.15.46:0/789350519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ba0108390 msgr2=0x7f8ba01a59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:41.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.776+0000 7f8ba5c3e700 1 --2- 172.21.15.46:0/789350519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ba0108390 0x7f8ba01a59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:41.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.776+0000 7f8ba5c3e700 1 -- 172.21.15.46:0/789350519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba01a5f30 con 0x7f8ba0102320 2024-04-26T21:33:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.776+0000 7f8ba643f700 1 --2- 172.21.15.46:0/789350519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ba0108390 0x7f8ba01a59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:33:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.776+0000 7f8ba5c3e700 1 --2- 172.21.15.46:0/789350519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ba0102320 0x7f8ba019a970 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c00b530 tx=0x7f8b9c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.776+0000 7f8b96ffd700 1 -- 172.21.15.46:0/789350519 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c015020 con 0x7f8ba0102320 2024-04-26T21:33:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.777+0000 7f8b96ffd700 1 -- 172.21.15.46:0/789350519 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b9c011ce0 con 0x7f8ba0102320 2024-04-26T21:33:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.777+0000 7f8ba7ea2700 1 -- 172.21.15.46:0/789350519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ba01a6160 con 0x7f8ba0102320 2024-04-26T21:33:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.777+0000 7f8b96ffd700 1 -- 172.21.15.46:0/789350519 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c010d20 con 0x7f8ba0102320 2024-04-26T21:33:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.777+0000 7f8ba7ea2700 1 -- 172.21.15.46:0/789350519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ba01989e0 con 0x7f8ba0102320 2024-04-26T21:33:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.778+0000 7f8ba7ea2700 1 -- 172.21.15.46:0/789350519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ba010a900 con 0x7f8ba0102320 2024-04-26T21:33:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.779+0000 7f8b96ffd700 1 -- 172.21.15.46:0/789350519 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8b9c018020 con 0x7f8ba0102320 2024-04-26T21:33:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.779+0000 7f8b96ffd700 1 --2- 172.21.15.46:0/789350519 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8b8c0a3d50 0x7f8b8c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.779+0000 7f8b96ffd700 1 -- 172.21.15.46:0/789350519 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f8b9c0cfc50 con 0x7f8ba0102320 2024-04-26T21:33:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.780+0000 7f8ba543d700 1 --2- 172.21.15.46:0/789350519 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8b8c0a3d50 0x7f8b8c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.780+0000 7f8ba543d700 1 --2- 172.21.15.46:0/789350519 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8b8c0a3d50 0x7f8b8c0a6210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8b90006010 tx=0x7f8b9001a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:41.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:40.785+0000 7f8b96ffd700 1 -- 172.21.15.46:0/789350519 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8b9c01f020 con 0x7f8ba0102320 2024-04-26T21:33:41.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:41.006+0000 7f8ba7ea2700 1 -- 172.21.15.46:0/789350519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f8ba010a2e0 con 0x7f8ba0102320 2024-04-26T21:33:41.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:41.006+0000 7f8b96ffd700 1 -- 172.21.15.46:0/789350519 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8b9c09c610 con 0x7f8ba0102320 2024-04-26T21:33:41.009 INFO:teuthology.orchestra.run.smithi046.stdout:219043332141 2024-04-26T21:33:41.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:41.008+0000 7f8ba7ea2700 1 -- 172.21.15.46:0/789350519 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8b8c0a3d50 msgr2=0x7f8b8c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:41.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:41.008+0000 7f8ba7ea2700 1 --2- 172.21.15.46:0/789350519 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8b8c0a3d50 0x7f8b8c0a6210 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8b90006010 tx=0x7f8b9001a040 comp rx=0 tx=0).stop 2024-04-26T21:33:41.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:41.009+0000 7f8ba7ea2700 1 -- 172.21.15.46:0/789350519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ba0102320 msgr2=0x7f8ba019a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:41.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:41.009+0000 7f8ba7ea2700 1 --2- 172.21.15.46:0/789350519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ba0102320 0x7f8ba019a970 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c00b530 tx=0x7f8b9c00b8f0 comp rx=0 tx=0).stop 2024-04-26T21:33:41.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:41.009+0000 7f8ba7ea2700 1 -- 172.21.15.46:0/789350519 shutdown_connections 2024-04-26T21:33:41.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:41.009+0000 7f8ba7ea2700 1 --2- 172.21.15.46:0/789350519 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8b8c0a3d50 0x7f8b8c0a6210 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:41.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:41.009+0000 7f8ba7ea2700 1 --2- 172.21.15.46:0/789350519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ba0102320 0x7f8ba019a970 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:41.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:41.009+0000 7f8ba7ea2700 1 --2- 172.21.15.46:0/789350519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ba0108390 0x7f8ba01a59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:41.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:41.009+0000 7f8ba7ea2700 1 --2- 172.21.15.46:0/789350519 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8ba0102ce0 0x7f8ba019aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:41.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:41.009+0000 7f8ba7ea2700 1 -- 172.21.15.46:0/789350519 >> 172.21.15.46:0/789350519 conn(0x7f8ba00792d0 msgr2=0x7f8ba0100f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:41.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:41.009+0000 7f8ba7ea2700 1 -- 172.21.15.46:0/789350519 shutdown_connections 2024-04-26T21:33:41.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:41.009+0000 7f8ba7ea2700 1 -- 172.21.15.46:0/789350519 wait complete. 2024-04-26T21:33:41.251 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:41 smithi046 ceph-mon[29201]: pgmap v322: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 11 B/s, 1 objects/s recovering 2024-04-26T21:33:41.251 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:41 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/789350519' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T21:33:41.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:41 smithi050 ceph-mon[32753]: pgmap v322: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 11 B/s, 1 objects/s recovering 2024-04-26T21:33:41.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:41 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/789350519' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T21:33:41.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:41 smithi142 ceph-mon[32944]: pgmap v322: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 11 B/s, 1 objects/s recovering 2024-04-26T21:33:41.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:41 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/789350519' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T21:33:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.366+0000 7fcb74538700 1 -- 172.21.15.46:0/599234377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb6c1025a0 msgr2=0x7fcb6c102980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.366+0000 7fcb74538700 1 --2- 172.21.15.46:0/599234377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb6c1025a0 0x7fcb6c102980 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcb680099c0 tx=0x7fcb68009cd0 comp rx=0 tx=0).stop 2024-04-26T21:33:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.366+0000 7fcb74538700 1 -- 172.21.15.46:0/599234377 shutdown_connections 2024-04-26T21:33:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.366+0000 7fcb74538700 1 --2- 172.21.15.46:0/599234377 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb6c1025a0 0x7fcb6c102980 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.366+0000 7fcb74538700 1 --2- 172.21.15.46:0/599234377 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb6c107680 0x7fcb6c10f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.366+0000 7fcb74538700 1 --2- 172.21.15.46:0/599234377 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb6c102f50 0x7fcb6c106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.366+0000 7fcb74538700 1 -- 172.21.15.46:0/599234377 >> 172.21.15.46:0/599234377 conn(0x7fcb6c0793c0 msgr2=0x7fcb6c0797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.366+0000 7fcb74538700 1 -- 172.21.15.46:0/599234377 shutdown_connections 2024-04-26T21:33:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.366+0000 7fcb74538700 1 -- 172.21.15.46:0/599234377 wait complete. 2024-04-26T21:33:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.366+0000 7fcb74538700 1 Processor -- start 2024-04-26T21:33:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.367+0000 7fcb74538700 1 -- start start 2024-04-26T21:33:42.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.367+0000 7fcb74538700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb6c1025a0 0x7fcb6c19f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:42.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.367+0000 7fcb74538700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb6c102f50 0x7fcb6c19fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:42.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.367+0000 7fcb74538700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb6c107680 0x7fcb6c199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:42.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.367+0000 7fcb74538700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c1125e0 con 0x7fcb6c107680 2024-04-26T21:33:42.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.367+0000 7fcb74538700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c112460 con 0x7fcb6c1025a0 2024-04-26T21:33:42.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.367+0000 7fcb74538700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c112760 con 0x7fcb6c102f50 2024-04-26T21:33:42.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.367+0000 7fcb72ad5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb6c107680 0x7fcb6c199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:42.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.367+0000 7fcb72ad5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb6c107680 0x7fcb6c199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43618/0 (socket says 172.21.15.46:43618) 2024-04-26T21:33:42.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.367+0000 7fcb72ad5700 1 -- 172.21.15.46:0/4181949074 learned_addr learned my addr 172.21.15.46:0/4181949074 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:33:42.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.367+0000 7fcb722d4700 1 --2- 172.21.15.46:0/4181949074 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb6c1025a0 0x7fcb6c19f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:42.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.367+0000 7fcb71ad3700 1 --2- 172.21.15.46:0/4181949074 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb6c102f50 0x7fcb6c19fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:42.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.367+0000 7fcb72ad5700 1 -- 172.21.15.46:0/4181949074 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb6c102f50 msgr2=0x7fcb6c19fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:42.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.367+0000 7fcb72ad5700 1 --2- 172.21.15.46:0/4181949074 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb6c102f50 0x7fcb6c19fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:42.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.367+0000 7fcb72ad5700 1 -- 172.21.15.46:0/4181949074 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb6c1025a0 msgr2=0x7fcb6c19f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:42.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.367+0000 7fcb72ad5700 1 --2- 172.21.15.46:0/4181949074 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb6c1025a0 0x7fcb6c19f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:42.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.367+0000 7fcb72ad5700 1 -- 172.21.15.46:0/4181949074 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb6c19a1d0 con 0x7fcb6c107680 2024-04-26T21:33:42.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.367+0000 7fcb722d4700 1 --2- 172.21.15.46:0/4181949074 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb6c1025a0 0x7fcb6c19f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:33:42.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.368+0000 7fcb72ad5700 1 --2- 172.21.15.46:0/4181949074 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb6c107680 0x7fcb6c199a70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcb64007ee0 tx=0x7fcb6400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:42.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.368+0000 7fcb637fe700 1 -- 172.21.15.46:0/4181949074 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb6400dbd0 con 0x7fcb6c107680 2024-04-26T21:33:42.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.368+0000 7fcb637fe700 1 -- 172.21.15.46:0/4181949074 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb6400dd70 con 0x7fcb6c107680 2024-04-26T21:33:42.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.368+0000 7fcb74538700 1 -- 172.21.15.46:0/4181949074 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb6c19a4c0 con 0x7fcb6c107680 2024-04-26T21:33:42.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.368+0000 7fcb637fe700 1 -- 172.21.15.46:0/4181949074 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb64005990 con 0x7fcb6c107680 2024-04-26T21:33:42.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.368+0000 7fcb74538700 1 -- 172.21.15.46:0/4181949074 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb6c1a6780 con 0x7fcb6c107680 2024-04-26T21:33:42.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.369+0000 7fcb74538700 1 -- 172.21.15.46:0/4181949074 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb6c06a110 con 0x7fcb6c107680 2024-04-26T21:33:42.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.370+0000 7fcb637fe700 1 -- 172.21.15.46:0/4181949074 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcb6401a020 con 0x7fcb6c107680 2024-04-26T21:33:42.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.371+0000 7fcb637fe700 1 --2- 172.21.15.46:0/4181949074 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcb580a3d50 0x7fcb580a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:42.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.371+0000 7fcb722d4700 1 --2- 172.21.15.46:0/4181949074 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcb580a3d50 0x7fcb580a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:42.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.371+0000 7fcb637fe700 1 -- 172.21.15.46:0/4181949074 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7fcb640d0410 con 0x7fcb6c107680 2024-04-26T21:33:42.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.372+0000 7fcb722d4700 1 --2- 172.21.15.46:0/4181949074 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcb580a3d50 0x7fcb580a6210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c19abb0 tx=0x7fcb68000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:42.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.375+0000 7fcb637fe700 1 -- 172.21.15.46:0/4181949074 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcb64010070 con 0x7fcb6c107680 2024-04-26T21:33:42.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.583+0000 7fcb74538700 1 -- 172.21.15.46:0/4181949074 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fcb6c1a6950 con 0x7fcb6c107680 2024-04-26T21:33:42.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.583+0000 7fcb637fe700 1 -- 172.21.15.46:0/4181949074 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcb6409cc60 con 0x7fcb6c107680 2024-04-26T21:33:42.585 INFO:teuthology.orchestra.run.smithi046.stdout:55834574925 2024-04-26T21:33:42.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.585+0000 7fcb74538700 1 -- 172.21.15.46:0/4181949074 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcb580a3d50 msgr2=0x7fcb580a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:42.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.585+0000 7fcb74538700 1 --2- 172.21.15.46:0/4181949074 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcb580a3d50 0x7fcb580a6210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c19abb0 tx=0x7fcb68000bc0 comp rx=0 tx=0).stop 2024-04-26T21:33:42.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.585+0000 7fcb74538700 1 -- 172.21.15.46:0/4181949074 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb6c107680 msgr2=0x7fcb6c199a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:42.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.585+0000 7fcb74538700 1 --2- 172.21.15.46:0/4181949074 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb6c107680 0x7fcb6c199a70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcb64007ee0 tx=0x7fcb6400d3b0 comp rx=0 tx=0).stop 2024-04-26T21:33:42.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.585+0000 7fcb74538700 1 -- 172.21.15.46:0/4181949074 shutdown_connections 2024-04-26T21:33:42.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.585+0000 7fcb74538700 1 --2- 172.21.15.46:0/4181949074 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcb580a3d50 0x7fcb580a6210 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:42.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.585+0000 7fcb74538700 1 --2- 172.21.15.46:0/4181949074 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb6c107680 0x7fcb6c199a70 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:42.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.585+0000 7fcb74538700 1 --2- 172.21.15.46:0/4181949074 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb6c1025a0 0x7fcb6c19f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:42.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.585+0000 7fcb74538700 1 --2- 172.21.15.46:0/4181949074 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcb6c102f50 0x7fcb6c19fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:42.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.585+0000 7fcb74538700 1 -- 172.21.15.46:0/4181949074 >> 172.21.15.46:0/4181949074 conn(0x7fcb6c0793c0 msgr2=0x7fcb6c101ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:42.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.586+0000 7fcb74538700 1 -- 172.21.15.46:0/4181949074 shutdown_connections 2024-04-26T21:33:42.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:42.586+0000 7fcb74538700 1 -- 172.21.15.46:0/4181949074 wait complete. 2024-04-26T21:33:43.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.049+0000 7f9d5fd5f700 1 -- 172.21.15.46:0/174951933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d58102320 msgr2=0x7f9d581027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:43.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.049+0000 7f9d5fd5f700 1 --2- 172.21.15.46:0/174951933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d58102320 0x7f9d581027a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9d48009990 tx=0x7f9d48009ca0 comp rx=0 tx=0).stop 2024-04-26T21:33:43.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.050+0000 7f9d5fd5f700 1 -- 172.21.15.46:0/174951933 shutdown_connections 2024-04-26T21:33:43.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.050+0000 7f9d5fd5f700 1 --2- 172.21.15.46:0/174951933 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d58102ce0 0x7f9d5810f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:43.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.050+0000 7f9d5fd5f700 1 --2- 172.21.15.46:0/174951933 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d58108390 0x7f9d58108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:43.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.050+0000 7f9d5fd5f700 1 --2- 172.21.15.46:0/174951933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d58102320 0x7f9d581027a0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:43.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.050+0000 7f9d5fd5f700 1 -- 172.21.15.46:0/174951933 >> 172.21.15.46:0/174951933 conn(0x7f9d580792d0 msgr2=0x7f9d580796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:43.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.050+0000 7f9d5fd5f700 1 -- 172.21.15.46:0/174951933 shutdown_connections 2024-04-26T21:33:43.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.050+0000 7f9d5fd5f700 1 -- 172.21.15.46:0/174951933 wait complete. 2024-04-26T21:33:43.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.050+0000 7f9d5fd5f700 1 Processor -- start 2024-04-26T21:33:43.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.050+0000 7f9d5fd5f700 1 -- start start 2024-04-26T21:33:43.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.051+0000 7f9d5fd5f700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d58102320 0x7f9d5819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:43.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.051+0000 7f9d5fd5f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d58102ce0 0x7f9d5819fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:43.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.051+0000 7f9d5fd5f700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d58108390 0x7f9d58199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:43.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.051+0000 7f9d5fd5f700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9d58112530 con 0x7f9d58102320 2024-04-26T21:33:43.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.051+0000 7f9d5fd5f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9d581123b0 con 0x7f9d58102ce0 2024-04-26T21:33:43.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.051+0000 7f9d5fd5f700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9d581126b0 con 0x7f9d58108390 2024-04-26T21:33:43.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.051+0000 7f9d5dafb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d58102320 0x7f9d5819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:43.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.051+0000 7f9d5dafb700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d58102320 0x7f9d5819f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:43624/0 (socket says 172.21.15.46:43624) 2024-04-26T21:33:43.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.051+0000 7f9d5dafb700 1 -- 172.21.15.46:0/2559581954 learned_addr learned my addr 172.21.15.46:0/2559581954 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:33:43.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.051+0000 7f9d5d2fa700 1 --2- 172.21.15.46:0/2559581954 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d58102ce0 0x7f9d5819fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:43.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.051+0000 7f9d5dafb700 1 -- 172.21.15.46:0/2559581954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d58108390 msgr2=0x7f9d58199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:43.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.051+0000 7f9d5dafb700 1 --2- 172.21.15.46:0/2559581954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d58108390 0x7f9d58199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:43.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.051+0000 7f9d5dafb700 1 -- 172.21.15.46:0/2559581954 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d58102ce0 msgr2=0x7f9d5819fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:43.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.051+0000 7f9d5dafb700 1 --2- 172.21.15.46:0/2559581954 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d58102ce0 0x7f9d5819fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:43.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.051+0000 7f9d5dafb700 1 -- 172.21.15.46:0/2559581954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d5819a0a0 con 0x7f9d58102320 2024-04-26T21:33:43.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.051+0000 7f9d5dafb700 1 --2- 172.21.15.46:0/2559581954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d58102320 0x7f9d5819f780 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9d5400b530 tx=0x7f9d5400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:43.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.052+0000 7f9d4effd700 1 -- 172.21.15.46:0/2559581954 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d54015020 con 0x7f9d58102320 2024-04-26T21:33:43.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.052+0000 7f9d4effd700 1 -- 172.21.15.46:0/2559581954 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d54011ce0 con 0x7f9d58102320 2024-04-26T21:33:43.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.052+0000 7f9d5fd5f700 1 -- 172.21.15.46:0/2559581954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d5819a390 con 0x7f9d58102320 2024-04-26T21:33:43.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.052+0000 7f9d4effd700 1 -- 172.21.15.46:0/2559581954 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d54010d20 con 0x7f9d58102320 2024-04-26T21:33:43.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.052+0000 7f9d5fd5f700 1 -- 172.21.15.46:0/2559581954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d581a6650 con 0x7f9d58102320 2024-04-26T21:33:43.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.053+0000 7f9d5fd5f700 1 -- 172.21.15.46:0/2559581954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d5806a110 con 0x7f9d58102320 2024-04-26T21:33:43.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.054+0000 7f9d4effd700 1 -- 172.21.15.46:0/2559581954 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9d54018020 con 0x7f9d58102320 2024-04-26T21:33:43.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.054+0000 7f9d4effd700 1 --2- 172.21.15.46:0/2559581954 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f9d440a3d50 0x7f9d440a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:43.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.055+0000 7f9d5d2fa700 1 --2- 172.21.15.46:0/2559581954 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f9d440a3d50 0x7f9d440a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:43.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.055+0000 7f9d4effd700 1 -- 172.21.15.46:0/2559581954 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f9d540d0170 con 0x7f9d58102320 2024-04-26T21:33:43.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.055+0000 7f9d5d2fa700 1 --2- 172.21.15.46:0/2559581954 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f9d440a3d50 0x7f9d440a6210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9d48006010 tx=0x7f9d4801a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:43.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.058+0000 7f9d4effd700 1 -- 172.21.15.46:0/2559581954 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9d5401f020 con 0x7f9d58102320 2024-04-26T21:33:43.067 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:33:43.281 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:43 smithi046 ceph-mon[29201]: pgmap v323: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 10 B/s, 1 objects/s recovering 2024-04-26T21:33:43.281 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:43 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/4181949074' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T21:33:43.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.280+0000 7f9d5fd5f700 1 -- 172.21.15.46:0/2559581954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f9d5819ae70 con 0x7f9d58102320 2024-04-26T21:33:43.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.280+0000 7f9d4effd700 1 -- 172.21.15.46:0/2559581954 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f9d5409cb30 con 0x7f9d58102320 2024-04-26T21:33:43.282 INFO:teuthology.orchestra.run.smithi046.stdout:287762808867 2024-04-26T21:33:43.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.283+0000 7f9d5fd5f700 1 -- 172.21.15.46:0/2559581954 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f9d440a3d50 msgr2=0x7f9d440a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:43.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.283+0000 7f9d5fd5f700 1 --2- 172.21.15.46:0/2559581954 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f9d440a3d50 0x7f9d440a6210 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9d48006010 tx=0x7f9d4801a040 comp rx=0 tx=0).stop 2024-04-26T21:33:43.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.283+0000 7f9d5fd5f700 1 -- 172.21.15.46:0/2559581954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d58102320 msgr2=0x7f9d5819f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:43.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.283+0000 7f9d5fd5f700 1 --2- 172.21.15.46:0/2559581954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d58102320 0x7f9d5819f780 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9d5400b530 tx=0x7f9d5400b8f0 comp rx=0 tx=0).stop 2024-04-26T21:33:43.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.283+0000 7f9d5fd5f700 1 -- 172.21.15.46:0/2559581954 shutdown_connections 2024-04-26T21:33:43.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.283+0000 7f9d5fd5f700 1 --2- 172.21.15.46:0/2559581954 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f9d440a3d50 0x7f9d440a6210 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:43.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.283+0000 7f9d5fd5f700 1 --2- 172.21.15.46:0/2559581954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d58102320 0x7f9d5819f780 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:43.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.283+0000 7f9d5fd5f700 1 --2- 172.21.15.46:0/2559581954 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9d58108390 0x7f9d58199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:43.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.283+0000 7f9d5fd5f700 1 --2- 172.21.15.46:0/2559581954 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d58102ce0 0x7f9d5819fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:43.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.283+0000 7f9d5fd5f700 1 -- 172.21.15.46:0/2559581954 >> 172.21.15.46:0/2559581954 conn(0x7f9d580792d0 msgr2=0x7f9d58106530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:43.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.283+0000 7f9d5fd5f700 1 -- 172.21.15.46:0/2559581954 shutdown_connections 2024-04-26T21:33:43.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:43.283+0000 7f9d5fd5f700 1 -- 172.21.15.46:0/2559581954 wait complete. 2024-04-26T21:33:43.345 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:43 smithi142 ceph-mon[32944]: pgmap v323: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 10 B/s, 1 objects/s recovering 2024-04-26T21:33:43.345 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:43 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/4181949074' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T21:33:43.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:43 smithi050 ceph-mon[32753]: pgmap v323: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 10 B/s, 1 objects/s recovering 2024-04-26T21:33:43.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:43 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/4181949074' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T21:33:43.512 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:33:43.512 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:33:43.538 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332138 got 219043332141 for osd.7 2024-04-26T21:33:43.538 DEBUG:teuthology.parallel:result is None 2024-04-26T21:33:44.202 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:44 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2559581954' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T21:33:44.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:44 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/2559581954' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T21:33:44.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:44 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/2559581954' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T21:33:44.618 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574922 got 55834574925 for osd.1 2024-04-26T21:33:44.619 DEBUG:teuthology.parallel:result is None 2024-04-26T21:33:45.131 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:45 smithi046 ceph-mon[29201]: pgmap v324: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 9 B/s, 0 objects/s recovering 2024-04-26T21:33:45.131 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:45 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:33:45.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:45 smithi142 ceph-mon[32944]: pgmap v324: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 9 B/s, 0 objects/s recovering 2024-04-26T21:33:45.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:45 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:33:45.363 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:45 smithi050 ceph-mon[32753]: pgmap v324: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 9 B/s, 0 objects/s recovering 2024-04-26T21:33:45.363 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:45 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:33:45.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.912+0000 7f4dfcb81700 1 -- 172.21.15.46:0/1312796183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df806bb50 msgr2=0x7f4df806bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:45.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.912+0000 7f4dfcb81700 1 --2- 172.21.15.46:0/1312796183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df806bb50 0x7f4df806bfd0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4dec0099c0 tx=0x7f4dec009cd0 comp rx=0 tx=0).stop 2024-04-26T21:33:45.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.912+0000 7f4dfcb81700 1 -- 172.21.15.46:0/1312796183 shutdown_connections 2024-04-26T21:33:45.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.912+0000 7f4dfcb81700 1 --2- 172.21.15.46:0/1312796183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4df806c510 0x7f4df81102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:45.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.912+0000 7f4dfcb81700 1 --2- 172.21.15.46:0/1312796183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4df8109510 0x7f4df81098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:45.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.912+0000 7f4dfcb81700 1 --2- 172.21.15.46:0/1312796183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df806bb50 0x7f4df806bfd0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:45.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.912+0000 7f4dfcb81700 1 -- 172.21.15.46:0/1312796183 >> 172.21.15.46:0/1312796183 conn(0x7f4df80790f0 msgr2=0x7f4df8079500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:45.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.912+0000 7f4dfcb81700 1 -- 172.21.15.46:0/1312796183 shutdown_connections 2024-04-26T21:33:45.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.912+0000 7f4dfcb81700 1 -- 172.21.15.46:0/1312796183 wait complete. 2024-04-26T21:33:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.913+0000 7f4dfcb81700 1 Processor -- start 2024-04-26T21:33:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.913+0000 7f4dfcb81700 1 -- start start 2024-04-26T21:33:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.913+0000 7f4dfcb81700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4df806bb50 0x7f4df8102710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.913+0000 7f4dfcb81700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df806c510 0x7f4df8102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.913+0000 7f4dfcb81700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4df8109510 0x7f4df8103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.913+0000 7f4dfcb81700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4df8113630 con 0x7f4df806c510 2024-04-26T21:33:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.913+0000 7f4dfcb81700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4df81134b0 con 0x7f4df8109510 2024-04-26T21:33:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.913+0000 7f4dfcb81700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4df81137b0 con 0x7f4df806bb50 2024-04-26T21:33:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.913+0000 7f4df5d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df806c510 0x7f4df8102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.913+0000 7f4df5d9b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df806c510 0x7f4df8102c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38632/0 (socket says 172.21.15.46:38632) 2024-04-26T21:33:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.913+0000 7f4df5d9b700 1 -- 172.21.15.46:0/4162795409 learned_addr learned my addr 172.21.15.46:0/4162795409 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:33:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.913+0000 7f4df659c700 1 --2- 172.21.15.46:0/4162795409 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4df806bb50 0x7f4df8102710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.913+0000 7f4df6d9d700 1 --2- 172.21.15.46:0/4162795409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4df8109510 0x7f4df8103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.914+0000 7f4df659c700 1 -- 172.21.15.46:0/4162795409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4df8109510 msgr2=0x7f4df8103190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.914+0000 7f4df659c700 1 --2- 172.21.15.46:0/4162795409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4df8109510 0x7f4df8103190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.914+0000 7f4df659c700 1 -- 172.21.15.46:0/4162795409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df806c510 msgr2=0x7f4df8102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.914+0000 7f4df659c700 1 --2- 172.21.15.46:0/4162795409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df806c510 0x7f4df8102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:45.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.914+0000 7f4df659c700 1 -- 172.21.15.46:0/4162795409 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df81a8010 con 0x7f4df806bb50 2024-04-26T21:33:45.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.914+0000 7f4df5d9b700 1 --2- 172.21.15.46:0/4162795409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df806c510 0x7f4df8102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:33:45.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.914+0000 7f4df659c700 1 --2- 172.21.15.46:0/4162795409 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4df806bb50 0x7f4df8102710 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4de800e9f0 tx=0x7f4de800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:45.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.914+0000 7f4de77fe700 1 -- 172.21.15.46:0/4162795409 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de8004220 con 0x7f4df806bb50 2024-04-26T21:33:45.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.914+0000 7f4de77fe700 1 -- 172.21.15.46:0/4162795409 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4de80043c0 con 0x7f4df806bb50 2024-04-26T21:33:45.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.914+0000 7f4dfcb81700 1 -- 172.21.15.46:0/4162795409 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4df81a8300 con 0x7f4df806bb50 2024-04-26T21:33:45.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.914+0000 7f4de77fe700 1 -- 172.21.15.46:0/4162795409 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de8005020 con 0x7f4df806bb50 2024-04-26T21:33:45.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.914+0000 7f4dfcb81700 1 -- 172.21.15.46:0/4162795409 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4df81a87f0 con 0x7f4df806bb50 2024-04-26T21:33:45.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.915+0000 7f4dfcb81700 1 -- 172.21.15.46:0/4162795409 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4df806a110 con 0x7f4df806bb50 2024-04-26T21:33:45.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.916+0000 7f4de77fe700 1 -- 172.21.15.46:0/4162795409 <== mon.1 v2:172.21.15.142:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4de8004750 con 0x7f4df806bb50 2024-04-26T21:33:45.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.917+0000 7f4de77fe700 1 --2- 172.21.15.46:0/4162795409 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4de00a3cd0 0x7f4de00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:45.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.917+0000 7f4df5d9b700 1 --2- 172.21.15.46:0/4162795409 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4de00a3cd0 0x7f4de00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:45.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.917+0000 7f4de77fe700 1 -- 172.21.15.46:0/4162795409 <== mon.1 v2:172.21.15.142:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f4de80d0450 con 0x7f4df806bb50 2024-04-26T21:33:45.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.917+0000 7f4df5d9b700 1 --2- 172.21.15.46:0/4162795409 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4de00a3cd0 0x7f4de00a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4df80796d0 tx=0x7f4dec005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:45.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:45.922+0000 7f4de77fe700 1 -- 172.21.15.46:0/4162795409 <== mon.1 v2:172.21.15.142:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4de809cd00 con 0x7f4df806bb50 2024-04-26T21:33:46.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.135+0000 7f4dfcb81700 1 -- 172.21.15.46:0/4162795409 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f4df8103c40 con 0x7f4df806bb50 2024-04-26T21:33:46.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.136+0000 7f4de77fe700 1 -- 172.21.15.46:0/4162795409 <== mon.1 v2:172.21.15.142:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4de809c600 con 0x7f4df806bb50 2024-04-26T21:33:46.137 INFO:teuthology.orchestra.run.smithi046.stdout:107374182467 2024-04-26T21:33:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.138+0000 7f4dfcb81700 1 -- 172.21.15.46:0/4162795409 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4de00a3cd0 msgr2=0x7f4de00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.138+0000 7f4dfcb81700 1 --2- 172.21.15.46:0/4162795409 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4de00a3cd0 0x7f4de00a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4df80796d0 tx=0x7f4dec005960 comp rx=0 tx=0).stop 2024-04-26T21:33:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.138+0000 7f4dfcb81700 1 -- 172.21.15.46:0/4162795409 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4df806bb50 msgr2=0x7f4df8102710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.138+0000 7f4dfcb81700 1 --2- 172.21.15.46:0/4162795409 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4df806bb50 0x7f4df8102710 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4de800e9f0 tx=0x7f4de800ed00 comp rx=0 tx=0).stop 2024-04-26T21:33:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.138+0000 7f4dfcb81700 1 -- 172.21.15.46:0/4162795409 shutdown_connections 2024-04-26T21:33:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.138+0000 7f4dfcb81700 1 --2- 172.21.15.46:0/4162795409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4df8109510 0x7f4df8103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.138+0000 7f4dfcb81700 1 --2- 172.21.15.46:0/4162795409 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4df806bb50 0x7f4df8102710 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.138+0000 7f4dfcb81700 1 --2- 172.21.15.46:0/4162795409 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4de00a3cd0 0x7f4de00a6190 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.138+0000 7f4dfcb81700 1 --2- 172.21.15.46:0/4162795409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4df806c510 0x7f4df8102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.138+0000 7f4dfcb81700 1 -- 172.21.15.46:0/4162795409 >> 172.21.15.46:0/4162795409 conn(0x7f4df80790f0 msgr2=0x7f4df81006f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.138+0000 7f4dfcb81700 1 -- 172.21.15.46:0/4162795409 shutdown_connections 2024-04-26T21:33:46.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.138+0000 7f4dfcb81700 1 -- 172.21.15.46:0/4162795409 wait complete. 2024-04-26T21:33:46.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.589+0000 7f7e19e5d700 1 -- 172.21.15.46:0/2942760294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e14103e90 msgr2=0x7f7e14110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:46.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.589+0000 7f7e19e5d700 1 --2- 172.21.15.46:0/2942760294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e14103e90 0x7f7e14110300 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c00b0d0 tx=0x7f7e0c00b3e0 comp rx=0 tx=0).stop 2024-04-26T21:33:46.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.589+0000 7f7e19e5d700 1 -- 172.21.15.46:0/2942760294 shutdown_connections 2024-04-26T21:33:46.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.589+0000 7f7e19e5d700 1 --2- 172.21.15.46:0/2942760294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e14103e90 0x7f7e14110300 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:46.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.589+0000 7f7e19e5d700 1 --2- 172.21.15.46:0/2942760294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e141083b0 0x7f7e14108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:46.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.589+0000 7f7e19e5d700 1 --2- 172.21.15.46:0/2942760294 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e141034d0 0x7f7e14103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:46.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.589+0000 7f7e19e5d700 1 -- 172.21.15.46:0/2942760294 >> 172.21.15.46:0/2942760294 conn(0x7f7e1407a580 msgr2=0x7f7e1407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:46.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.590+0000 7f7e19e5d700 1 -- 172.21.15.46:0/2942760294 shutdown_connections 2024-04-26T21:33:46.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.590+0000 7f7e19e5d700 1 -- 172.21.15.46:0/2942760294 wait complete. 2024-04-26T21:33:46.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.590+0000 7f7e19e5d700 1 Processor -- start 2024-04-26T21:33:46.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.590+0000 7f7e19e5d700 1 -- start start 2024-04-26T21:33:46.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.590+0000 7f7e19e5d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e141034d0 0x7f7e14075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:46.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.590+0000 7f7e19e5d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e14103e90 0x7f7e14075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:46.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.590+0000 7f7e19e5d700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e141083b0 0x7f7e140761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:46.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.590+0000 7f7e19e5d700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7e14113620 con 0x7f7e141034d0 2024-04-26T21:33:46.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.590+0000 7f7e19e5d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7e141134a0 con 0x7f7e14103e90 2024-04-26T21:33:46.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.590+0000 7f7e19e5d700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7e141137a0 con 0x7f7e141083b0 2024-04-26T21:33:46.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.590+0000 7f7e137fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e141034d0 0x7f7e14075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:46.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.591+0000 7f7e137fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e141034d0 0x7f7e14075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38646/0 (socket says 172.21.15.46:38646) 2024-04-26T21:33:46.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.591+0000 7f7e137fe700 1 -- 172.21.15.46:0/3772142808 learned_addr learned my addr 172.21.15.46:0/3772142808 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:33:46.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.591+0000 7f7e0bfff700 1 --2- 172.21.15.46:0/3772142808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e14103e90 0x7f7e14075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:46.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.591+0000 7f7e13fff700 1 --2- 172.21.15.46:0/3772142808 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e141083b0 0x7f7e140761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:46.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.591+0000 7f7e137fe700 1 -- 172.21.15.46:0/3772142808 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e141083b0 msgr2=0x7f7e140761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:46.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.591+0000 7f7e137fe700 1 --2- 172.21.15.46:0/3772142808 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e141083b0 0x7f7e140761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:46.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.591+0000 7f7e137fe700 1 -- 172.21.15.46:0/3772142808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e14103e90 msgr2=0x7f7e14075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:46.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.591+0000 7f7e137fe700 1 --2- 172.21.15.46:0/3772142808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e14103e90 0x7f7e14075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:46.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.591+0000 7f7e137fe700 1 -- 172.21.15.46:0/3772142808 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e141a7f00 con 0x7f7e141034d0 2024-04-26T21:33:46.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.591+0000 7f7e137fe700 1 --2- 172.21.15.46:0/3772142808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e141034d0 0x7f7e14075720 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7e0400c9b0 tx=0x7f7e0400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:46.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.591+0000 7f7e117fa700 1 -- 172.21.15.46:0/3772142808 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e04007b00 con 0x7f7e141034d0 2024-04-26T21:33:46.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.591+0000 7f7e19e5d700 1 -- 172.21.15.46:0/3772142808 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e141a81f0 con 0x7f7e141034d0 2024-04-26T21:33:46.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.591+0000 7f7e117fa700 1 -- 172.21.15.46:0/3772142808 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7e0400ce50 con 0x7f7e141034d0 2024-04-26T21:33:46.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.591+0000 7f7e19e5d700 1 -- 172.21.15.46:0/3772142808 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e141a8780 con 0x7f7e141034d0 2024-04-26T21:33:46.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.591+0000 7f7e117fa700 1 -- 172.21.15.46:0/3772142808 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e04018660 con 0x7f7e141034d0 2024-04-26T21:33:46.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.592+0000 7f7e19e5d700 1 -- 172.21.15.46:0/3772142808 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e1406a110 con 0x7f7e141034d0 2024-04-26T21:33:46.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.594+0000 7f7e117fa700 1 -- 172.21.15.46:0/3772142808 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7e0400f450 con 0x7f7e141034d0 2024-04-26T21:33:46.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.594+0000 7f7e117fa700 1 --2- 172.21.15.46:0/3772142808 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7e000a3cd0 0x7f7e000a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:46.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.594+0000 7f7e0bfff700 1 --2- 172.21.15.46:0/3772142808 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7e000a3cd0 0x7f7e000a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:46.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.594+0000 7f7e117fa700 1 -- 172.21.15.46:0/3772142808 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f7e040d06c0 con 0x7f7e141034d0 2024-04-26T21:33:46.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.595+0000 7f7e0bfff700 1 --2- 172.21.15.46:0/3772142808 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7e000a3cd0 0x7f7e000a6190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7e1407ab10 tx=0x7f7dfc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:46.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.598+0000 7f7e117fa700 1 -- 172.21.15.46:0/3772142808 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7e04014030 con 0x7f7e141034d0 2024-04-26T21:33:46.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.811+0000 7f7e19e5d700 1 -- 172.21.15.46:0/3772142808 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f7e14076c50 con 0x7f7e141034d0 2024-04-26T21:33:46.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.811+0000 7f7e117fa700 1 -- 172.21.15.46:0/3772142808 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7e0409d000 con 0x7f7e141034d0 2024-04-26T21:33:46.813 INFO:teuthology.orchestra.run.smithi046.stdout:34359738450 2024-04-26T21:33:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.813+0000 7f7e19e5d700 1 -- 172.21.15.46:0/3772142808 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7e000a3cd0 msgr2=0x7f7e000a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.813+0000 7f7e19e5d700 1 --2- 172.21.15.46:0/3772142808 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7e000a3cd0 0x7f7e000a6190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7e1407ab10 tx=0x7f7dfc009260 comp rx=0 tx=0).stop 2024-04-26T21:33:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.813+0000 7f7e19e5d700 1 -- 172.21.15.46:0/3772142808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e141034d0 msgr2=0x7f7e14075720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.813+0000 7f7e19e5d700 1 --2- 172.21.15.46:0/3772142808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e141034d0 0x7f7e14075720 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7e0400c9b0 tx=0x7f7e0400ccc0 comp rx=0 tx=0).stop 2024-04-26T21:33:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.813+0000 7f7e19e5d700 1 -- 172.21.15.46:0/3772142808 shutdown_connections 2024-04-26T21:33:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.813+0000 7f7e19e5d700 1 --2- 172.21.15.46:0/3772142808 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7e000a3cd0 0x7f7e000a6190 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.813+0000 7f7e19e5d700 1 --2- 172.21.15.46:0/3772142808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e141034d0 0x7f7e14075720 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.814+0000 7f7e19e5d700 1 --2- 172.21.15.46:0/3772142808 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7e141083b0 0x7f7e140761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.814+0000 7f7e19e5d700 1 --2- 172.21.15.46:0/3772142808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e14103e90 0x7f7e14075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.814+0000 7f7e19e5d700 1 -- 172.21.15.46:0/3772142808 >> 172.21.15.46:0/3772142808 conn(0x7f7e1407a580 msgr2=0x7f7e141014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.814+0000 7f7e19e5d700 1 -- 172.21.15.46:0/3772142808 shutdown_connections 2024-04-26T21:33:46.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:46.814+0000 7f7e19e5d700 1 -- 172.21.15.46:0/3772142808 wait complete. 2024-04-26T21:33:47.023 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182463 got 107374182467 for osd.3 2024-04-26T21:33:47.023 DEBUG:teuthology.parallel:result is None 2024-04-26T21:33:47.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:46 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:33:47.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:46 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:33:47.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:46 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:33:47.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:46 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:33:47.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:46 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/4162795409' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T21:33:47.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:46 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3772142808' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T21:33:47.151 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:46 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:33:47.151 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:46 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:33:47.151 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:46 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:33:47.151 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:46 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:33:47.151 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:46 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/4162795409' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T21:33:47.151 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:46 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3772142808' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T21:33:47.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:46 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:33:47.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:46 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:33:47.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:46 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:33:47.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:46 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:33:47.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:46 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/4162795409' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T21:33:47.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:46 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3772142808' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T21:33:48.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:47 smithi046 ceph-mon[29201]: pgmap v325: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s; 9 B/s, 0 objects/s recovering 2024-04-26T21:33:48.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:47 smithi050 ceph-mon[32753]: pgmap v325: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s; 9 B/s, 0 objects/s recovering 2024-04-26T21:33:48.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:47 smithi142 ceph-mon[32944]: pgmap v325: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s; 9 B/s, 0 objects/s recovering 2024-04-26T21:33:48.497 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:33:48.498 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:33:48.504 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808864 got 287762808867 for osd.9 2024-04-26T21:33:48.504 DEBUG:teuthology.parallel:result is None 2024-04-26T21:33:48.973 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738445 got 34359738450 for osd.0 2024-04-26T21:33:48.974 DEBUG:teuthology.parallel:result is None 2024-04-26T21:33:49.799 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:33:50.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:49 smithi050 ceph-mon[32753]: pgmap v326: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:50.142 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:49 smithi046 ceph-mon[29201]: pgmap v326: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:50.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:49 smithi142 ceph-mon[32944]: pgmap v326: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:51.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.596+0000 7fde5e960700 1 -- 172.21.15.46:0/1351960071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde581094a0 msgr2=0x7fde58109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.596+0000 7fde5e960700 1 --2- 172.21.15.46:0/1351960071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde581094a0 0x7fde58109880 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fde48009990 tx=0x7fde48009ca0 comp rx=0 tx=0).stop 2024-04-26T21:33:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.596+0000 7fde5e960700 1 -- 172.21.15.46:0/1351960071 shutdown_connections 2024-04-26T21:33:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.596+0000 7fde5e960700 1 --2- 172.21.15.46:0/1351960071 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde581094a0 0x7fde58109880 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.596+0000 7fde5e960700 1 --2- 172.21.15.46:0/1351960071 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5806bad0 0x7fde581102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.596+0000 7fde5e960700 1 --2- 172.21.15.46:0/1351960071 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde5806b130 0x7fde5806b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.596+0000 7fde5e960700 1 -- 172.21.15.46:0/1351960071 >> 172.21.15.46:0/1351960071 conn(0x7fde580792d0 msgr2=0x7fde580796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.596+0000 7fde5e960700 1 -- 172.21.15.46:0/1351960071 shutdown_connections 2024-04-26T21:33:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.596+0000 7fde5e960700 1 -- 172.21.15.46:0/1351960071 wait complete. 2024-04-26T21:33:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.597+0000 7fde5e960700 1 Processor -- start 2024-04-26T21:33:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.597+0000 7fde5e960700 1 -- start start 2024-04-26T21:33:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.597+0000 7fde5e960700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5806b130 0x7fde5819f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.597+0000 7fde5e960700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde5806bad0 0x7fde5819fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.597+0000 7fde5e960700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde581094a0 0x7fde581999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:51.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.597+0000 7fde5e960700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fde581135e0 con 0x7fde581094a0 2024-04-26T21:33:51.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.597+0000 7fde5e960700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fde58113460 con 0x7fde5806b130 2024-04-26T21:33:51.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.597+0000 7fde5e960700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fde58113760 con 0x7fde5806bad0 2024-04-26T21:33:51.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.597+0000 7fde5cefd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde581094a0 0x7fde581999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:51.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.597+0000 7fde5cefd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde581094a0 0x7fde581999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38650/0 (socket says 172.21.15.46:38650) 2024-04-26T21:33:51.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.597+0000 7fde5cefd700 1 -- 172.21.15.46:0/3289009411 learned_addr learned my addr 172.21.15.46:0/3289009411 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:33:51.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.597+0000 7fde577fe700 1 --2- 172.21.15.46:0/3289009411 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde5806bad0 0x7fde5819fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:51.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.597+0000 7fde57fff700 1 --2- 172.21.15.46:0/3289009411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5806b130 0x7fde5819f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:51.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.597+0000 7fde5cefd700 1 -- 172.21.15.46:0/3289009411 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde5806bad0 msgr2=0x7fde5819fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:51.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.597+0000 7fde5cefd700 1 --2- 172.21.15.46:0/3289009411 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde5806bad0 0x7fde5819fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:51.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.597+0000 7fde5cefd700 1 -- 172.21.15.46:0/3289009411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5806b130 msgr2=0x7fde5819f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:51.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.597+0000 7fde5cefd700 1 --2- 172.21.15.46:0/3289009411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5806b130 0x7fde5819f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:51.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.597+0000 7fde5cefd700 1 -- 172.21.15.46:0/3289009411 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde5819a100 con 0x7fde581094a0 2024-04-26T21:33:51.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.598+0000 7fde5cefd700 1 --2- 172.21.15.46:0/3289009411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde581094a0 0x7fde581999a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fde50007ee0 tx=0x7fde5000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:51.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.598+0000 7fde557fa700 1 -- 172.21.15.46:0/3289009411 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5000dbd0 con 0x7fde581094a0 2024-04-26T21:33:51.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.598+0000 7fde5e960700 1 -- 172.21.15.46:0/3289009411 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde5819a3f0 con 0x7fde581094a0 2024-04-26T21:33:51.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.598+0000 7fde557fa700 1 -- 172.21.15.46:0/3289009411 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde5000dd70 con 0x7fde581094a0 2024-04-26T21:33:51.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.598+0000 7fde557fa700 1 -- 172.21.15.46:0/3289009411 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde50005990 con 0x7fde581094a0 2024-04-26T21:33:51.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.598+0000 7fde5e960700 1 -- 172.21.15.46:0/3289009411 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde581a66b0 con 0x7fde581094a0 2024-04-26T21:33:51.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.599+0000 7fde5e960700 1 -- 172.21.15.46:0/3289009411 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde5806a110 con 0x7fde581094a0 2024-04-26T21:33:51.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.600+0000 7fde557fa700 1 -- 172.21.15.46:0/3289009411 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fde5001a020 con 0x7fde581094a0 2024-04-26T21:33:51.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.601+0000 7fde557fa700 1 --2- 172.21.15.46:0/3289009411 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fde400a3cd0 0x7fde400a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:51.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.601+0000 7fde57fff700 1 --2- 172.21.15.46:0/3289009411 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fde400a3cd0 0x7fde400a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:51.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.601+0000 7fde557fa700 1 -- 172.21.15.46:0/3289009411 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7fde500cfcc0 con 0x7fde581094a0 2024-04-26T21:33:51.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.602+0000 7fde57fff700 1 --2- 172.21.15.46:0/3289009411 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fde400a3cd0 0x7fde400a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fde4800b5c0 tx=0x7fde48005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:51.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.606+0000 7fde557fa700 1 -- 172.21.15.46:0/3289009411 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fde50010070 con 0x7fde581094a0 2024-04-26T21:33:51.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.823+0000 7fde5e960700 1 -- 172.21.15.46:0/3289009411 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fde5802e490 con 0x7fde581094a0 2024-04-26T21:33:51.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.823+0000 7fde557fa700 1 -- 172.21.15.46:0/3289009411 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fde5009c570 con 0x7fde581094a0 2024-04-26T21:33:51.824 INFO:teuthology.orchestra.run.smithi046.stdout:343597383707 2024-04-26T21:33:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.825+0000 7fde5e960700 1 -- 172.21.15.46:0/3289009411 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fde400a3cd0 msgr2=0x7fde400a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.825+0000 7fde5e960700 1 --2- 172.21.15.46:0/3289009411 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fde400a3cd0 0x7fde400a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fde4800b5c0 tx=0x7fde48005fb0 comp rx=0 tx=0).stop 2024-04-26T21:33:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.826+0000 7fde5e960700 1 -- 172.21.15.46:0/3289009411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde581094a0 msgr2=0x7fde581999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.826+0000 7fde5e960700 1 --2- 172.21.15.46:0/3289009411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde581094a0 0x7fde581999a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fde50007ee0 tx=0x7fde5000d3b0 comp rx=0 tx=0).stop 2024-04-26T21:33:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.826+0000 7fde5e960700 1 -- 172.21.15.46:0/3289009411 shutdown_connections 2024-04-26T21:33:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.826+0000 7fde5e960700 1 --2- 172.21.15.46:0/3289009411 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fde400a3cd0 0x7fde400a6190 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.826+0000 7fde5e960700 1 --2- 172.21.15.46:0/3289009411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde581094a0 0x7fde581999a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.826+0000 7fde5e960700 1 --2- 172.21.15.46:0/3289009411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde5806b130 0x7fde5819f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.826+0000 7fde5e960700 1 --2- 172.21.15.46:0/3289009411 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fde5806bad0 0x7fde5819fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.826+0000 7fde5e960700 1 -- 172.21.15.46:0/3289009411 >> 172.21.15.46:0/3289009411 conn(0x7fde580792d0 msgr2=0x7fde581004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.826+0000 7fde5e960700 1 -- 172.21.15.46:0/3289009411 shutdown_connections 2024-04-26T21:33:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:51.826+0000 7fde5e960700 1 -- 172.21.15.46:0/3289009411 wait complete. 2024-04-26T21:33:52.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:51 smithi046 ceph-mon[29201]: pgmap v327: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:33:52.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:51 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3289009411' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T21:33:52.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:51 smithi050 ceph-mon[32753]: pgmap v327: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:33:52.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:51 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3289009411' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T21:33:52.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:51 smithi142 ceph-mon[32944]: pgmap v327: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:33:52.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:51 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3289009411' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T21:33:53.609 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383701 got 343597383707 for osd.11 2024-04-26T21:33:53.609 DEBUG:teuthology.parallel:result is None 2024-04-26T21:33:54.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:53 smithi050 ceph-mon[32753]: pgmap v328: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:54.173 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:53 smithi046 ceph-mon[29201]: pgmap v328: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:54.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:53 smithi142 ceph-mon[32944]: pgmap v328: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:54.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.897+0000 7f4ec3a6a700 1 -- 172.21.15.46:0/606757001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ebc103470 msgr2=0x7f4ebc1038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:54.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.897+0000 7f4ec3a6a700 1 --2- 172.21.15.46:0/606757001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ebc103470 0x7f4ebc1038f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8009990 tx=0x7f4eb8009ca0 comp rx=0 tx=0).stop 2024-04-26T21:33:54.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.897+0000 7f4ec3a6a700 1 -- 172.21.15.46:0/606757001 shutdown_connections 2024-04-26T21:33:54.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.897+0000 7f4ec3a6a700 1 --2- 172.21.15.46:0/606757001 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ebc103e30 0x7f4ebc110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:54.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.897+0000 7f4ec3a6a700 1 --2- 172.21.15.46:0/606757001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4ebc1094c0 0x7f4ebc1098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:54.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.897+0000 7f4ec3a6a700 1 --2- 172.21.15.46:0/606757001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ebc103470 0x7f4ebc1038f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:54.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.897+0000 7f4ec3a6a700 1 -- 172.21.15.46:0/606757001 >> 172.21.15.46:0/606757001 conn(0x7f4ebc07a580 msgr2=0x7f4ebc07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:54.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.897+0000 7f4ec3a6a700 1 -- 172.21.15.46:0/606757001 shutdown_connections 2024-04-26T21:33:54.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.897+0000 7f4ec3a6a700 1 -- 172.21.15.46:0/606757001 wait complete. 2024-04-26T21:33:54.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.898+0000 7f4ec3a6a700 1 Processor -- start 2024-04-26T21:33:54.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.898+0000 7f4ec3a6a700 1 -- start start 2024-04-26T21:33:54.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.898+0000 7f4ec3a6a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ebc103470 0x7f4ebc075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:54.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.898+0000 7f4ec3a6a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4ebc103e30 0x7f4ebc075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:54.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.898+0000 7f4ec3a6a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ebc1094c0 0x7f4ebc0761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:54.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.898+0000 7f4ec3a6a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4ebc1135f0 con 0x7f4ebc103470 2024-04-26T21:33:54.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.898+0000 7f4ec3a6a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4ebc113470 con 0x7f4ebc103e30 2024-04-26T21:33:54.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.898+0000 7f4ec3a6a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4ebc113770 con 0x7f4ebc1094c0 2024-04-26T21:33:54.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.898+0000 7f4ec1806700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ebc103470 0x7f4ebc075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:54.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.899+0000 7f4ec1806700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ebc103470 0x7f4ebc075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38666/0 (socket says 172.21.15.46:38666) 2024-04-26T21:33:54.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.899+0000 7f4ec1806700 1 -- 172.21.15.46:0/1028841760 learned_addr learned my addr 172.21.15.46:0/1028841760 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:33:54.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.899+0000 7f4ec2007700 1 --2- 172.21.15.46:0/1028841760 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ebc1094c0 0x7f4ebc0761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:54.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.899+0000 7f4ec1005700 1 --2- 172.21.15.46:0/1028841760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4ebc103e30 0x7f4ebc075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:54.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.899+0000 7f4ec1806700 1 -- 172.21.15.46:0/1028841760 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ebc1094c0 msgr2=0x7f4ebc0761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:54.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.899+0000 7f4ec1806700 1 --2- 172.21.15.46:0/1028841760 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ebc1094c0 0x7f4ebc0761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:54.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.899+0000 7f4ec1806700 1 -- 172.21.15.46:0/1028841760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4ebc103e30 msgr2=0x7f4ebc075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:54.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.899+0000 7f4ec1806700 1 --2- 172.21.15.46:0/1028841760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4ebc103e30 0x7f4ebc075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:54.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.899+0000 7f4ec1806700 1 -- 172.21.15.46:0/1028841760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ebc1a8170 con 0x7f4ebc103470 2024-04-26T21:33:54.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.899+0000 7f4ec2007700 1 --2- 172.21.15.46:0/1028841760 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ebc1094c0 0x7f4ebc0761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:33:54.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.899+0000 7f4ec1005700 1 --2- 172.21.15.46:0/1028841760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4ebc103e30 0x7f4ebc075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:33:54.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.899+0000 7f4ec1806700 1 --2- 172.21.15.46:0/1028841760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ebc103470 0x7f4ebc075720 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4eac00d980 tx=0x7f4eac00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:54.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.899+0000 7f4eb2ffd700 1 -- 172.21.15.46:0/1028841760 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eac016070 con 0x7f4ebc103470 2024-04-26T21:33:54.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.899+0000 7f4eb2ffd700 1 -- 172.21.15.46:0/1028841760 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4eac0044d0 con 0x7f4ebc103470 2024-04-26T21:33:54.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.899+0000 7f4ec3a6a700 1 -- 172.21.15.46:0/1028841760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ebc1a8460 con 0x7f4ebc103470 2024-04-26T21:33:54.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.899+0000 7f4eb2ffd700 1 -- 172.21.15.46:0/1028841760 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eac004d00 con 0x7f4ebc103470 2024-04-26T21:33:54.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.899+0000 7f4ec3a6a700 1 -- 172.21.15.46:0/1028841760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ebc1a89f0 con 0x7f4ebc103470 2024-04-26T21:33:54.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.900+0000 7f4ec3a6a700 1 -- 172.21.15.46:0/1028841760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ebc10b930 con 0x7f4ebc103470 2024-04-26T21:33:54.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.902+0000 7f4eb2ffd700 1 -- 172.21.15.46:0/1028841760 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4eac004020 con 0x7f4ebc103470 2024-04-26T21:33:54.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.902+0000 7f4eb2ffd700 1 --2- 172.21.15.46:0/1028841760 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4ea80a3d50 0x7f4ea80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:54.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.902+0000 7f4ec1005700 1 --2- 172.21.15.46:0/1028841760 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4ea80a3d50 0x7f4ea80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:54.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.902+0000 7f4eb2ffd700 1 -- 172.21.15.46:0/1028841760 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f4eac0d0330 con 0x7f4ebc103470 2024-04-26T21:33:54.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.903+0000 7f4ec1005700 1 --2- 172.21.15.46:0/1028841760 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4ea80a3d50 0x7f4ea80a6210 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8006010 tx=0x7f4eb801a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:54.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:54.909+0000 7f4eb2ffd700 1 -- 172.21.15.46:0/1028841760 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4eac01f020 con 0x7f4ebc103470 2024-04-26T21:33:55.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:55.125+0000 7f4ec3a6a700 1 -- 172.21.15.46:0/1028841760 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f4ebc076c50 con 0x7f4ebc103470 2024-04-26T21:33:55.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:55.126+0000 7f4eb2ffd700 1 -- 172.21.15.46:0/1028841760 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4eac09cbe0 con 0x7f4ebc103470 2024-04-26T21:33:55.127 INFO:teuthology.orchestra.run.smithi046.stdout:197568495669 2024-04-26T21:33:55.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:55.128+0000 7f4ec3a6a700 1 -- 172.21.15.46:0/1028841760 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4ea80a3d50 msgr2=0x7f4ea80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:55.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:55.128+0000 7f4ec3a6a700 1 --2- 172.21.15.46:0/1028841760 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4ea80a3d50 0x7f4ea80a6210 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8006010 tx=0x7f4eb801a040 comp rx=0 tx=0).stop 2024-04-26T21:33:55.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:55.128+0000 7f4ec3a6a700 1 -- 172.21.15.46:0/1028841760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ebc103470 msgr2=0x7f4ebc075720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:55.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:55.128+0000 7f4ec3a6a700 1 --2- 172.21.15.46:0/1028841760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ebc103470 0x7f4ebc075720 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4eac00d980 tx=0x7f4eac00dd40 comp rx=0 tx=0).stop 2024-04-26T21:33:55.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:55.128+0000 7f4ec3a6a700 1 -- 172.21.15.46:0/1028841760 shutdown_connections 2024-04-26T21:33:55.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:55.128+0000 7f4ec3a6a700 1 --2- 172.21.15.46:0/1028841760 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4ea80a3d50 0x7f4ea80a6210 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:55.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:55.128+0000 7f4ec3a6a700 1 --2- 172.21.15.46:0/1028841760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ebc103470 0x7f4ebc075720 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:55.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:55.128+0000 7f4ec3a6a700 1 --2- 172.21.15.46:0/1028841760 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4ebc1094c0 0x7f4ebc0761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:55.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:55.128+0000 7f4ec3a6a700 1 --2- 172.21.15.46:0/1028841760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4ebc103e30 0x7f4ebc075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:55.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:55.128+0000 7f4ec3a6a700 1 -- 172.21.15.46:0/1028841760 >> 172.21.15.46:0/1028841760 conn(0x7f4ebc07a580 msgr2=0x7f4ebc1022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:55.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:55.128+0000 7f4ec3a6a700 1 -- 172.21.15.46:0/1028841760 shutdown_connections 2024-04-26T21:33:55.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:55.128+0000 7f4ec3a6a700 1 -- 172.21.15.46:0/1028841760 wait complete. 2024-04-26T21:33:55.879 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495664 got 197568495669 for osd.6 2024-04-26T21:33:55.879 DEBUG:teuthology.parallel:result is None 2024-04-26T21:33:56.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.079+0000 7f6aeff9a700 1 -- 172.21.15.46:0/2955288637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae8101a10 msgr2=0x7f6ae8101e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:56.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.079+0000 7f6aeff9a700 1 --2- 172.21.15.46:0/2955288637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae8101a10 0x7f6ae8101e70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8009990 tx=0x7f6ad8009ca0 comp rx=0 tx=0).stop 2024-04-26T21:33:56.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.079+0000 7f6aeff9a700 1 -- 172.21.15.46:0/2955288637 shutdown_connections 2024-04-26T21:33:56.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.079+0000 7f6aeff9a700 1 --2- 172.21.15.46:0/2955288637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6ae81023b0 0x7f6ae81102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:56.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.079+0000 7f6aeff9a700 1 --2- 172.21.15.46:0/2955288637 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ae81094b0 0x7f6ae8109890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:56.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.079+0000 7f6aeff9a700 1 --2- 172.21.15.46:0/2955288637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae8101a10 0x7f6ae8101e70 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:56.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.079+0000 7f6aeff9a700 1 -- 172.21.15.46:0/2955288637 >> 172.21.15.46:0/2955288637 conn(0x7f6ae8078990 msgr2=0x7f6ae80ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:56.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.079+0000 7f6aeff9a700 1 -- 172.21.15.46:0/2955288637 shutdown_connections 2024-04-26T21:33:56.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.079+0000 7f6aeff9a700 1 -- 172.21.15.46:0/2955288637 wait complete. 2024-04-26T21:33:56.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.080+0000 7f6aeff9a700 1 Processor -- start 2024-04-26T21:33:56.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.080+0000 7f6aeff9a700 1 -- start start 2024-04-26T21:33:56.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.080+0000 7f6aeff9a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ae8101a10 0x7f6ae819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:56.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.080+0000 7f6aeff9a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae81023b0 0x7f6ae819fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:56.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.080+0000 7f6aeff9a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6ae81094b0 0x7f6ae8199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:56.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.080+0000 7f6aeff9a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae81135c0 con 0x7f6ae81023b0 2024-04-26T21:33:56.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.080+0000 7f6aeff9a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae8113440 con 0x7f6ae81094b0 2024-04-26T21:33:56.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.080+0000 7f6aeff9a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae8113740 con 0x7f6ae8101a10 2024-04-26T21:33:56.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.080+0000 7f6aed535700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae81023b0 0x7f6ae819fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:56.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.080+0000 7f6aed535700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae81023b0 0x7f6ae819fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41754/0 (socket says 172.21.15.46:41754) 2024-04-26T21:33:56.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.080+0000 7f6aed535700 1 -- 172.21.15.46:0/1266076817 learned_addr learned my addr 172.21.15.46:0/1266076817 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:33:56.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.080+0000 7f6aed535700 1 -- 172.21.15.46:0/1266076817 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ae8101a10 msgr2=0x7f6ae819f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:56.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.080+0000 7f6aee537700 1 --2- 172.21.15.46:0/1266076817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6ae81094b0 0x7f6ae8199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:56.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.080+0000 7f6aedd36700 1 --2- 172.21.15.46:0/1266076817 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ae8101a10 0x7f6ae819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:56.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.080+0000 7f6aed535700 1 --2- 172.21.15.46:0/1266076817 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ae8101a10 0x7f6ae819f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:56.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.080+0000 7f6aed535700 1 -- 172.21.15.46:0/1266076817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6ae81094b0 msgr2=0x7f6ae8199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:56.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.080+0000 7f6aed535700 1 --2- 172.21.15.46:0/1266076817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6ae81094b0 0x7f6ae8199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:56.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.080+0000 7f6aed535700 1 -- 172.21.15.46:0/1266076817 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ae819a0a0 con 0x7f6ae81023b0 2024-04-26T21:33:56.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.080+0000 7f6aee537700 1 --2- 172.21.15.46:0/1266076817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6ae81094b0 0x7f6ae8199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:33:56.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.080+0000 7f6aed535700 1 --2- 172.21.15.46:0/1266076817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae81023b0 0x7f6ae819fcc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8000c00 tx=0x7f6ad80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:56.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.081+0000 7f6adeffd700 1 -- 172.21.15.46:0/1266076817 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad8027070 con 0x7f6ae81023b0 2024-04-26T21:33:56.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.081+0000 7f6adeffd700 1 -- 172.21.15.46:0/1266076817 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ad8003fa0 con 0x7f6ae81023b0 2024-04-26T21:33:56.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.081+0000 7f6aeff9a700 1 -- 172.21.15.46:0/1266076817 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ae819a330 con 0x7f6ae81023b0 2024-04-26T21:33:56.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.081+0000 7f6adeffd700 1 -- 172.21.15.46:0/1266076817 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad8022db0 con 0x7f6ae81023b0 2024-04-26T21:33:56.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.081+0000 7f6aeff9a700 1 -- 172.21.15.46:0/1266076817 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ae81a6650 con 0x7f6ae81023b0 2024-04-26T21:33:56.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.082+0000 7f6aeff9a700 1 -- 172.21.15.46:0/1266076817 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ae8104d30 con 0x7f6ae81023b0 2024-04-26T21:33:56.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.083+0000 7f6adeffd700 1 -- 172.21.15.46:0/1266076817 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6ad802a070 con 0x7f6ae81023b0 2024-04-26T21:33:56.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.084+0000 7f6adeffd700 1 --2- 172.21.15.46:0/1266076817 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6ad40a3cd0 0x7f6ad40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:56.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.084+0000 7f6aedd36700 1 --2- 172.21.15.46:0/1266076817 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6ad40a3cd0 0x7f6ad40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:56.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.084+0000 7f6adeffd700 1 -- 172.21.15.46:0/1266076817 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f6ad80d1650 con 0x7f6ae81023b0 2024-04-26T21:33:56.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.084+0000 7f6aedd36700 1 --2- 172.21.15.46:0/1266076817 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6ad40a3cd0 0x7f6ad40a6190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6ae819aa80 tx=0x7f6ae4005ee0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:56.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.087+0000 7f6adeffd700 1 -- 172.21.15.46:0/1266076817 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6ad8013070 con 0x7f6ae81023b0 2024-04-26T21:33:56.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:55 smithi050 ceph-mon[32753]: pgmap v329: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:56.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:55 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/1028841760' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T21:33:56.173 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:55 smithi046 ceph-mon[29201]: pgmap v329: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:56.173 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:55 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1028841760' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T21:33:56.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:55 smithi142 ceph-mon[32944]: pgmap v329: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:56.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:55 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/1028841760' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T21:33:56.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.298+0000 7f6aeff9a700 1 -- 172.21.15.46:0/1266076817 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f6ae819ae70 con 0x7f6ae81023b0 2024-04-26T21:33:56.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.299+0000 7f6adeffd700 1 -- 172.21.15.46:0/1266076817 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6ad809df90 con 0x7f6ae81023b0 2024-04-26T21:33:56.300 INFO:teuthology.orchestra.run.smithi046.stdout:141733920832 2024-04-26T21:33:56.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.302+0000 7f6aeff9a700 1 -- 172.21.15.46:0/1266076817 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6ad40a3cd0 msgr2=0x7f6ad40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:56.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.302+0000 7f6aeff9a700 1 --2- 172.21.15.46:0/1266076817 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6ad40a3cd0 0x7f6ad40a6190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6ae819aa80 tx=0x7f6ae4005ee0 comp rx=0 tx=0).stop 2024-04-26T21:33:56.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.302+0000 7f6aeff9a700 1 -- 172.21.15.46:0/1266076817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae81023b0 msgr2=0x7f6ae819fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:56.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.302+0000 7f6aeff9a700 1 --2- 172.21.15.46:0/1266076817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae81023b0 0x7f6ae819fcc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8000c00 tx=0x7f6ad80047b0 comp rx=0 tx=0).stop 2024-04-26T21:33:56.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.302+0000 7f6aeff9a700 1 -- 172.21.15.46:0/1266076817 shutdown_connections 2024-04-26T21:33:56.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.302+0000 7f6aeff9a700 1 --2- 172.21.15.46:0/1266076817 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6ae81094b0 0x7f6ae8199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:56.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.302+0000 7f6aeff9a700 1 --2- 172.21.15.46:0/1266076817 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6ae8101a10 0x7f6ae819f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:56.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.302+0000 7f6aeff9a700 1 --2- 172.21.15.46:0/1266076817 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6ad40a3cd0 0x7f6ad40a6190 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:56.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.302+0000 7f6aeff9a700 1 --2- 172.21.15.46:0/1266076817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ae81023b0 0x7f6ae819fcc0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:56.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.302+0000 7f6aeff9a700 1 -- 172.21.15.46:0/1266076817 >> 172.21.15.46:0/1266076817 conn(0x7f6ae8078990 msgr2=0x7f6ae8106450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:56.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.302+0000 7f6aeff9a700 1 -- 172.21.15.46:0/1266076817 shutdown_connections 2024-04-26T21:33:56.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:56.302+0000 7f6aeff9a700 1 -- 172.21.15.46:0/1266076817 wait complete. 2024-04-26T21:33:57.023 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:56 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1266076817' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T21:33:57.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:56 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/1266076817' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T21:33:57.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:56 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/1266076817' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T21:33:57.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.448+0000 7fcec13fa700 1 -- 172.21.15.46:0/2645966281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcebc06c700 msgr2=0x7fcebc1080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:57.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.448+0000 7fcec13fa700 1 --2- 172.21.15.46:0/2645966281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcebc06c700 0x7fcebc1080c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fcea40099c0 tx=0x7fcea4009cd0 comp rx=0 tx=0).stop 2024-04-26T21:33:57.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.448+0000 7fcec13fa700 1 -- 172.21.15.46:0/2645966281 shutdown_connections 2024-04-26T21:33:57.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.448+0000 7fcec13fa700 1 --2- 172.21.15.46:0/2645966281 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcebc1087f0 0x7fcebc110370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:57.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.448+0000 7fcec13fa700 1 --2- 172.21.15.46:0/2645966281 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcebc06bd50 0x7fcebc06c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:57.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.448+0000 7fcec13fa700 1 --2- 172.21.15.46:0/2645966281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcebc06c700 0x7fcebc1080c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:57.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.448+0000 7fcec13fa700 1 -- 172.21.15.46:0/2645966281 >> 172.21.15.46:0/2645966281 conn(0x7fcebc0792d0 msgr2=0x7fcebc0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:57.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.449+0000 7fcec13fa700 1 -- 172.21.15.46:0/2645966281 shutdown_connections 2024-04-26T21:33:57.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.449+0000 7fcec13fa700 1 -- 172.21.15.46:0/2645966281 wait complete. 2024-04-26T21:33:57.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.449+0000 7fcec13fa700 1 Processor -- start 2024-04-26T21:33:57.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.450+0000 7fcec13fa700 1 -- start start 2024-04-26T21:33:57.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.450+0000 7fcec13fa700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcebc06bd50 0x7fcebc19f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:57.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.450+0000 7fcec13fa700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcebc06c700 0x7fcebc19fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:57.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.450+0000 7fcec13fa700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcebc1087f0 0x7fcebc1998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:57.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.450+0000 7fcec13fa700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc113630 con 0x7fcebc06bd50 2024-04-26T21:33:57.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.450+0000 7fcec13fa700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc1134b0 con 0x7fcebc06c700 2024-04-26T21:33:57.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.450+0000 7fcec13fa700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcebc1137b0 con 0x7fcebc1087f0 2024-04-26T21:33:57.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.450+0000 7fcebaffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcebc06bd50 0x7fcebc19f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:57.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.450+0000 7fceba7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcebc06c700 0x7fcebc19fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:57.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.450+0000 7fcebaffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcebc06bd50 0x7fcebc19f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41766/0 (socket says 172.21.15.46:41766) 2024-04-26T21:33:57.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.450+0000 7fcebaffd700 1 -- 172.21.15.46:0/440094486 learned_addr learned my addr 172.21.15.46:0/440094486 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:33:57.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.450+0000 7fcebb7fe700 1 --2- 172.21.15.46:0/440094486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcebc1087f0 0x7fcebc1998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:57.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.451+0000 7fceba7fc700 1 -- 172.21.15.46:0/440094486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcebc1087f0 msgr2=0x7fcebc1998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:57.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.451+0000 7fceba7fc700 1 --2- 172.21.15.46:0/440094486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcebc1087f0 0x7fcebc1998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:57.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.451+0000 7fceba7fc700 1 -- 172.21.15.46:0/440094486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcebc06bd50 msgr2=0x7fcebc19f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:57.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.451+0000 7fceba7fc700 1 --2- 172.21.15.46:0/440094486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcebc06bd50 0x7fcebc19f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:57.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.451+0000 7fceba7fc700 1 -- 172.21.15.46:0/440094486 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcebc19a050 con 0x7fcebc06c700 2024-04-26T21:33:57.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.451+0000 7fcebaffd700 1 --2- 172.21.15.46:0/440094486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcebc06bd50 0x7fcebc19f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:33:57.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.451+0000 7fceba7fc700 1 --2- 172.21.15.46:0/440094486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcebc06c700 0x7fcebc19fc70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcea4009f50 tx=0x7fcea40047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:57.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.451+0000 7fceb3fff700 1 -- 172.21.15.46:0/440094486 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea4027070 con 0x7fcebc06c700 2024-04-26T21:33:57.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.451+0000 7fceb3fff700 1 -- 172.21.15.46:0/440094486 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcea4003f20 con 0x7fcebc06c700 2024-04-26T21:33:57.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.451+0000 7fcec13fa700 1 -- 172.21.15.46:0/440094486 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcebc19a2e0 con 0x7fcebc06c700 2024-04-26T21:33:57.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.451+0000 7fceb3fff700 1 -- 172.21.15.46:0/440094486 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea4022bd0 con 0x7fcebc06c700 2024-04-26T21:33:57.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.451+0000 7fcec13fa700 1 -- 172.21.15.46:0/440094486 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcebc1a6600 con 0x7fcebc06c700 2024-04-26T21:33:57.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.452+0000 7fcec13fa700 1 -- 172.21.15.46:0/440094486 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcebc06b3a0 con 0x7fcebc06c700 2024-04-26T21:33:57.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.454+0000 7fceb3fff700 1 -- 172.21.15.46:0/440094486 <== mon.2 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcea402a070 con 0x7fcebc06c700 2024-04-26T21:33:57.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.455+0000 7fceb3fff700 1 --2- 172.21.15.46:0/440094486 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcea80a3cd0 0x7fcea80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:57.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.455+0000 7fcebaffd700 1 --2- 172.21.15.46:0/440094486 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcea80a3cd0 0x7fcea80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:57.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.455+0000 7fceb3fff700 1 -- 172.21.15.46:0/440094486 <== mon.2 v2:172.21.15.50:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7fcea40d1850 con 0x7fcebc06c700 2024-04-26T21:33:57.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.455+0000 7fcebaffd700 1 --2- 172.21.15.46:0/440094486 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcea80a3cd0 0x7fcea80a6190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcebc19aa30 tx=0x7fceac00b500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:57.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.459+0000 7fceb3fff700 1 -- 172.21.15.46:0/440094486 <== mon.2 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcea4031070 con 0x7fcebc06c700 2024-04-26T21:33:57.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.674+0000 7fcec13fa700 1 -- 172.21.15.46:0/440094486 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fcebc104b70 con 0x7fcebc06c700 2024-04-26T21:33:57.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.675+0000 7fceb3fff700 1 -- 172.21.15.46:0/440094486 <== mon.2 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fcea4013070 con 0x7fcebc06c700 2024-04-26T21:33:57.676 INFO:teuthology.orchestra.run.smithi046.stdout:253403070508 2024-04-26T21:33:57.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.677+0000 7fcec13fa700 1 -- 172.21.15.46:0/440094486 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcea80a3cd0 msgr2=0x7fcea80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:57.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.677+0000 7fcec13fa700 1 --2- 172.21.15.46:0/440094486 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcea80a3cd0 0x7fcea80a6190 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcebc19aa30 tx=0x7fceac00b500 comp rx=0 tx=0).stop 2024-04-26T21:33:57.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.677+0000 7fcec13fa700 1 -- 172.21.15.46:0/440094486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcebc06c700 msgr2=0x7fcebc19fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:57.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.677+0000 7fcec13fa700 1 --2- 172.21.15.46:0/440094486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcebc06c700 0x7fcebc19fc70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcea4009f50 tx=0x7fcea40047b0 comp rx=0 tx=0).stop 2024-04-26T21:33:57.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.677+0000 7fcec13fa700 1 -- 172.21.15.46:0/440094486 shutdown_connections 2024-04-26T21:33:57.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.677+0000 7fcec13fa700 1 --2- 172.21.15.46:0/440094486 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcea80a3cd0 0x7fcea80a6190 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:57.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.677+0000 7fcec13fa700 1 --2- 172.21.15.46:0/440094486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcebc06bd50 0x7fcebc19f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:57.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.677+0000 7fcec13fa700 1 --2- 172.21.15.46:0/440094486 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcebc1087f0 0x7fcebc1998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:57.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.677+0000 7fcec13fa700 1 --2- 172.21.15.46:0/440094486 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcebc06c700 0x7fcebc19fc70 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:57.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.677+0000 7fcec13fa700 1 -- 172.21.15.46:0/440094486 >> 172.21.15.46:0/440094486 conn(0x7fcebc0792d0 msgr2=0x7fcebc100a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:57.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.677+0000 7fcec13fa700 1 -- 172.21.15.46:0/440094486 shutdown_connections 2024-04-26T21:33:57.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:57.677+0000 7fcec13fa700 1 -- 172.21.15.46:0/440094486 wait complete. 2024-04-26T21:33:58.048 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:57 smithi046 ceph-mon[29201]: pgmap v330: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:33:58.048 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:57 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/440094486' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T21:33:58.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:57 smithi050 ceph-mon[32753]: pgmap v330: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:33:58.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:57 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/440094486' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T21:33:58.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:57 smithi142 ceph-mon[32944]: pgmap v330: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:33:58.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:57 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/440094486' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T21:33:58.735 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070500 got 253403070508 for osd.8 2024-04-26T21:33:58.736 DEBUG:teuthology.parallel:result is None 2024-04-26T21:33:59.006 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:58 smithi046 ceph-mon[29201]: pgmap v331: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:59.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.004+0000 7f31eb34e700 1 -- 172.21.15.46:0/2660178373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31e41094a0 msgr2=0x7f31e4109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:59.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.004+0000 7f31eb34e700 1 --2- 172.21.15.46:0/2660178373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31e41094a0 0x7f31e4109880 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f31e0009990 tx=0x7f31e0009ca0 comp rx=0 tx=0).stop 2024-04-26T21:33:59.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.004+0000 7f31eb34e700 1 -- 172.21.15.46:0/2660178373 shutdown_connections 2024-04-26T21:33:59.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.004+0000 7f31eb34e700 1 --2- 172.21.15.46:0/2660178373 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31e41094a0 0x7f31e4109880 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:59.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.004+0000 7f31eb34e700 1 --2- 172.21.15.46:0/2660178373 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31e406bad0 0x7f31e41102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:59.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.004+0000 7f31eb34e700 1 --2- 172.21.15.46:0/2660178373 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31e406b130 0x7f31e406b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:59.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.004+0000 7f31eb34e700 1 -- 172.21.15.46:0/2660178373 >> 172.21.15.46:0/2660178373 conn(0x7f31e40792d0 msgr2=0x7f31e40796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:59.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.005+0000 7f31eb34e700 1 -- 172.21.15.46:0/2660178373 shutdown_connections 2024-04-26T21:33:59.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.005+0000 7f31eb34e700 1 -- 172.21.15.46:0/2660178373 wait complete. 2024-04-26T21:33:59.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.006+0000 7f31eb34e700 1 Processor -- start 2024-04-26T21:33:59.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.006+0000 7f31eb34e700 1 -- start start 2024-04-26T21:33:59.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.006+0000 7f31eb34e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31e406b130 0x7f31e419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:59.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.006+0000 7f31eb34e700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31e406bad0 0x7f31e419fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:59.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.006+0000 7f31eb34e700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31e41094a0 0x7f31e41998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:59.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.006+0000 7f31eb34e700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f31e41135b0 con 0x7f31e406bad0 2024-04-26T21:33:59.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.006+0000 7f31eb34e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f31e4113430 con 0x7f31e406b130 2024-04-26T21:33:59.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.006+0000 7f31eb34e700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f31e4113730 con 0x7f31e41094a0 2024-04-26T21:33:59.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.006+0000 7f31e88e9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31e406bad0 0x7f31e419fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:59.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.006+0000 7f31e88e9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31e406bad0 0x7f31e419fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41790/0 (socket says 172.21.15.46:41790) 2024-04-26T21:33:59.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.006+0000 7f31e88e9700 1 -- 172.21.15.46:0/3349345245 learned_addr learned my addr 172.21.15.46:0/3349345245 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:33:59.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.006+0000 7f31e98eb700 1 --2- 172.21.15.46:0/3349345245 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31e41094a0 0x7f31e41998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:59.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.006+0000 7f31e90ea700 1 --2- 172.21.15.46:0/3349345245 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31e406b130 0x7f31e419f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:59.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.006+0000 7f31e88e9700 1 -- 172.21.15.46:0/3349345245 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31e41094a0 msgr2=0x7f31e41998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:59.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.006+0000 7f31e88e9700 1 --2- 172.21.15.46:0/3349345245 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31e41094a0 0x7f31e41998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:59.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.006+0000 7f31e88e9700 1 -- 172.21.15.46:0/3349345245 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31e406b130 msgr2=0x7f31e419f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:59.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.006+0000 7f31e88e9700 1 --2- 172.21.15.46:0/3349345245 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31e406b130 0x7f31e419f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:59.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.006+0000 7f31e88e9700 1 -- 172.21.15.46:0/3349345245 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31e419a050 con 0x7f31e406bad0 2024-04-26T21:33:59.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.007+0000 7f31e88e9700 1 --2- 172.21.15.46:0/3349345245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31e406bad0 0x7f31e419fc70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f31d400d980 tx=0x7f31d400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:59.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.007+0000 7f31da7fc700 1 -- 172.21.15.46:0/3349345245 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d4017070 con 0x7f31e406bad0 2024-04-26T21:33:59.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.007+0000 7f31eb34e700 1 -- 172.21.15.46:0/3349345245 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31e419a340 con 0x7f31e406bad0 2024-04-26T21:33:59.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.007+0000 7f31da7fc700 1 -- 172.21.15.46:0/3349345245 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31d4005020 con 0x7f31e406bad0 2024-04-26T21:33:59.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.007+0000 7f31da7fc700 1 -- 172.21.15.46:0/3349345245 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d4009b50 con 0x7f31e406bad0 2024-04-26T21:33:59.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.007+0000 7f31eb34e700 1 -- 172.21.15.46:0/3349345245 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31e41a6600 con 0x7f31e406bad0 2024-04-26T21:33:59.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.008+0000 7f31eb34e700 1 -- 172.21.15.46:0/3349345245 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31e406a110 con 0x7f31e406bad0 2024-04-26T21:33:59.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.009+0000 7f31da7fc700 1 -- 172.21.15.46:0/3349345245 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f31d40051d0 con 0x7f31e406bad0 2024-04-26T21:33:59.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.010+0000 7f31da7fc700 1 --2- 172.21.15.46:0/3349345245 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f31d00a3cd0 0x7f31d00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:59.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.010+0000 7f31e90ea700 1 --2- 172.21.15.46:0/3349345245 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f31d00a3cd0 0x7f31d00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:59.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.010+0000 7f31da7fc700 1 -- 172.21.15.46:0/3349345245 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f31d40d08e0 con 0x7f31e406bad0 2024-04-26T21:33:59.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.010+0000 7f31e90ea700 1 --2- 172.21.15.46:0/3349345245 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f31d00a3cd0 0x7f31d00a6190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f31e0006010 tx=0x7f31e000b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:59.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.013+0000 7f31da7fc700 1 -- 172.21.15.46:0/3349345245 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f31d4020020 con 0x7f31e406bad0 2024-04-26T21:33:59.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:58 smithi050 ceph-mon[32753]: pgmap v331: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:59.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.223+0000 7f31eb34e700 1 -- 172.21.15.46:0/3349345245 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f31e402e490 con 0x7f31e406bad0 2024-04-26T21:33:59.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.223+0000 7f31da7fc700 1 -- 172.21.15.46:0/3349345245 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f31d409d130 con 0x7f31e406bad0 2024-04-26T21:33:59.224 INFO:teuthology.orchestra.run.smithi046.stdout:77309411402 2024-04-26T21:33:59.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.225+0000 7f31eb34e700 1 -- 172.21.15.46:0/3349345245 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f31d00a3cd0 msgr2=0x7f31d00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:59.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.225+0000 7f31eb34e700 1 --2- 172.21.15.46:0/3349345245 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f31d00a3cd0 0x7f31d00a6190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f31e0006010 tx=0x7f31e000b540 comp rx=0 tx=0).stop 2024-04-26T21:33:59.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.225+0000 7f31eb34e700 1 -- 172.21.15.46:0/3349345245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31e406bad0 msgr2=0x7f31e419fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:59.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.225+0000 7f31eb34e700 1 --2- 172.21.15.46:0/3349345245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31e406bad0 0x7f31e419fc70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f31d400d980 tx=0x7f31d400dc90 comp rx=0 tx=0).stop 2024-04-26T21:33:59.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.225+0000 7f31eb34e700 1 -- 172.21.15.46:0/3349345245 shutdown_connections 2024-04-26T21:33:59.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.225+0000 7f31eb34e700 1 --2- 172.21.15.46:0/3349345245 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f31e41094a0 0x7f31e41998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:59.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.225+0000 7f31eb34e700 1 --2- 172.21.15.46:0/3349345245 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31e406b130 0x7f31e419f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:59.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.225+0000 7f31eb34e700 1 --2- 172.21.15.46:0/3349345245 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f31d00a3cd0 0x7f31d00a6190 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:59.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.225+0000 7f31eb34e700 1 --2- 172.21.15.46:0/3349345245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f31e406bad0 0x7f31e419fc70 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:59.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.225+0000 7f31eb34e700 1 -- 172.21.15.46:0/3349345245 >> 172.21.15.46:0/3349345245 conn(0x7f31e40792d0 msgr2=0x7f31e4100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:59.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.226+0000 7f31eb34e700 1 -- 172.21.15.46:0/3349345245 shutdown_connections 2024-04-26T21:33:59.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.226+0000 7f31eb34e700 1 -- 172.21.15.46:0/3349345245 wait complete. 2024-04-26T21:33:59.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:58 smithi142 ceph-mon[32944]: pgmap v331: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:33:59.611 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920825 got 141733920832 for osd.4 2024-04-26T21:33:59.612 DEBUG:teuthology.parallel:result is None 2024-04-26T21:33:59.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.807+0000 7f64a1412700 1 -- 172.21.15.46:0/201677175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f649c075650 msgr2=0x7f649c075a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:59.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.807+0000 7f64a1412700 1 --2- 172.21.15.46:0/201677175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f649c075650 0x7f649c075a30 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f648c009990 tx=0x7f648c009ca0 comp rx=0 tx=0).stop 2024-04-26T21:33:59.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.807+0000 7f64a1412700 1 -- 172.21.15.46:0/201677175 shutdown_connections 2024-04-26T21:33:59.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.807+0000 7f64a1412700 1 --2- 172.21.15.46:0/201677175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f649c075650 0x7f649c075a30 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:59.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.807+0000 7f64a1412700 1 --2- 172.21.15.46:0/201677175 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f649c10fe20 0x7f649c112210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:59.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.807+0000 7f64a1412700 1 --2- 172.21.15.46:0/201677175 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f649c075f70 0x7f649c10f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:59.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.807+0000 7f64a1412700 1 -- 172.21.15.46:0/201677175 >> 172.21.15.46:0/201677175 conn(0x7f649c07a580 msgr2=0x7f649c07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:59.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.807+0000 7f64a1412700 1 -- 172.21.15.46:0/201677175 shutdown_connections 2024-04-26T21:33:59.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.807+0000 7f64a1412700 1 -- 172.21.15.46:0/201677175 wait complete. 2024-04-26T21:33:59.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.808+0000 7f64a1412700 1 Processor -- start 2024-04-26T21:33:59.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.808+0000 7f64a1412700 1 -- start start 2024-04-26T21:33:59.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.808+0000 7f64a1412700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f649c075650 0x7f649c19bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:59.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.808+0000 7f64a1412700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f649c075f70 0x7f649c19c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:59.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.808+0000 7f64a1412700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f649c10fe20 0x7f649c1a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:59.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.808+0000 7f64a1412700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f649c1157c0 con 0x7f649c10fe20 2024-04-26T21:33:59.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.808+0000 7f64a1412700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f649c115640 con 0x7f649c075f70 2024-04-26T21:33:59.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.808+0000 7f64a1412700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f649c115940 con 0x7f649c075650 2024-04-26T21:33:59.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.808+0000 7f649b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f649c10fe20 0x7f649c1a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:59.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.808+0000 7f649b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f649c10fe20 0x7f649c1a05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41804/0 (socket says 172.21.15.46:41804) 2024-04-26T21:33:59.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.808+0000 7f649b7fe700 1 -- 172.21.15.46:0/4235675096 learned_addr learned my addr 172.21.15.46:0/4235675096 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:33:59.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.808+0000 7f6493fff700 1 --2- 172.21.15.46:0/4235675096 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f649c075f70 0x7f649c19c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:59.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.808+0000 7f649affd700 1 --2- 172.21.15.46:0/4235675096 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f649c075650 0x7f649c19bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:59.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.809+0000 7f649b7fe700 1 -- 172.21.15.46:0/4235675096 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f649c075650 msgr2=0x7f649c19bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:59.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.809+0000 7f649b7fe700 1 --2- 172.21.15.46:0/4235675096 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f649c075650 0x7f649c19bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:59.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.809+0000 7f649b7fe700 1 -- 172.21.15.46:0/4235675096 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f649c075f70 msgr2=0x7f649c19c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:59.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.809+0000 7f649b7fe700 1 --2- 172.21.15.46:0/4235675096 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f649c075f70 0x7f649c19c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:59.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.809+0000 7f649b7fe700 1 -- 172.21.15.46:0/4235675096 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f649c1a0d30 con 0x7f649c10fe20 2024-04-26T21:33:59.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.809+0000 7f649b7fe700 1 --2- 172.21.15.46:0/4235675096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f649c10fe20 0x7f649c1a05f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6494007d70 tx=0x7f649400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:59.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.809+0000 7f6498ff9700 1 -- 172.21.15.46:0/4235675096 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6494016020 con 0x7f649c10fe20 2024-04-26T21:33:59.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.809+0000 7f6498ff9700 1 -- 172.21.15.46:0/4235675096 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f649400ad40 con 0x7f649c10fe20 2024-04-26T21:33:59.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.809+0000 7f64a1412700 1 -- 172.21.15.46:0/4235675096 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f649c1a1020 con 0x7f649c10fe20 2024-04-26T21:33:59.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.809+0000 7f6498ff9700 1 -- 172.21.15.46:0/4235675096 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64940115e0 con 0x7f649c10fe20 2024-04-26T21:33:59.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.809+0000 7f64a1412700 1 -- 172.21.15.46:0/4235675096 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f649c1a88a0 con 0x7f649c10fe20 2024-04-26T21:33:59.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.810+0000 7f64a1412700 1 -- 172.21.15.46:0/4235675096 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f649c06a110 con 0x7f649c10fe20 2024-04-26T21:33:59.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.811+0000 7f6498ff9700 1 -- 172.21.15.46:0/4235675096 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6494019020 con 0x7f649c10fe20 2024-04-26T21:33:59.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.812+0000 7f6498ff9700 1 --2- 172.21.15.46:0/4235675096 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f64880a3cd0 0x7f64880a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:59.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.812+0000 7f649affd700 1 --2- 172.21.15.46:0/4235675096 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f64880a3cd0 0x7f64880a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:59.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.812+0000 7f6498ff9700 1 -- 172.21.15.46:0/4235675096 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f64940d08d0 con 0x7f649c10fe20 2024-04-26T21:33:59.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.812+0000 7f649affd700 1 --2- 172.21.15.46:0/4235675096 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f64880a3cd0 0x7f64880a6190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f648c006010 tx=0x7f648c01a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:59.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:33:59.816+0000 7f6498ff9700 1 -- 172.21.15.46:0/4235675096 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6494020020 con 0x7f649c10fe20 2024-04-26T21:34:00.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:00.025+0000 7f64a1412700 1 -- 172.21.15.46:0/4235675096 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f649c02e490 con 0x7f649c10fe20 2024-04-26T21:34:00.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:00.025+0000 7f6498ff9700 1 -- 172.21.15.46:0/4235675096 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f649409d230 con 0x7f649c10fe20 2024-04-26T21:34:00.026 INFO:teuthology.orchestra.run.smithi046.stdout:163208757308 2024-04-26T21:34:00.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:00.027+0000 7f64a1412700 1 -- 172.21.15.46:0/4235675096 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f64880a3cd0 msgr2=0x7f64880a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:00.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:00.027+0000 7f64a1412700 1 --2- 172.21.15.46:0/4235675096 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f64880a3cd0 0x7f64880a6190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f648c006010 tx=0x7f648c01a040 comp rx=0 tx=0).stop 2024-04-26T21:34:00.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:00.027+0000 7f64a1412700 1 -- 172.21.15.46:0/4235675096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f649c10fe20 msgr2=0x7f649c1a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:00.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:00.027+0000 7f64a1412700 1 --2- 172.21.15.46:0/4235675096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f649c10fe20 0x7f649c1a05f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6494007d70 tx=0x7f649400a340 comp rx=0 tx=0).stop 2024-04-26T21:34:00.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:00.027+0000 7f64a1412700 1 -- 172.21.15.46:0/4235675096 shutdown_connections 2024-04-26T21:34:00.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:00.027+0000 7f64a1412700 1 --2- 172.21.15.46:0/4235675096 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f64880a3cd0 0x7f64880a6190 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:00.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:00.027+0000 7f64a1412700 1 --2- 172.21.15.46:0/4235675096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f649c10fe20 0x7f649c1a05f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:00.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:00.027+0000 7f64a1412700 1 --2- 172.21.15.46:0/4235675096 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f649c075650 0x7f649c19bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:00.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:00.027+0000 7f64a1412700 1 --2- 172.21.15.46:0/4235675096 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f649c075f70 0x7f649c19c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:00.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:00.027+0000 7f64a1412700 1 -- 172.21.15.46:0/4235675096 >> 172.21.15.46:0/4235675096 conn(0x7f649c07a580 msgr2=0x7f649c101970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:00.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:00.027+0000 7f64a1412700 1 -- 172.21.15.46:0/4235675096 shutdown_connections 2024-04-26T21:34:00.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:00.027+0000 7f64a1412700 1 -- 172.21.15.46:0/4235675096 wait complete. 2024-04-26T21:34:00.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:33:59 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3349345245' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T21:34:00.118 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:33:59 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3349345245' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T21:34:00.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:33:59 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3349345245' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T21:34:00.802 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757302 got 163208757308 for osd.5 2024-04-26T21:34:00.803 DEBUG:teuthology.parallel:result is None 2024-04-26T21:34:01.034 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:00 smithi046 ceph-mon[29201]: pgmap v332: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:34:01.034 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:00 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/4235675096' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T21:34:01.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.264+0000 7f8b9072a700 1 -- 172.21.15.46:0/999363851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b88109490 msgr2=0x7f8b88109870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:01.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.264+0000 7f8b9072a700 1 --2- 172.21.15.46:0/999363851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b88109490 0x7f8b88109870 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8b84009990 tx=0x7f8b84009ca0 comp rx=0 tx=0).stop 2024-04-26T21:34:01.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.264+0000 7f8b9072a700 1 -- 172.21.15.46:0/999363851 shutdown_connections 2024-04-26T21:34:01.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.264+0000 7f8b9072a700 1 --2- 172.21.15.46:0/999363851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b88109490 0x7f8b88109870 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:01.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.264+0000 7f8b9072a700 1 --2- 172.21.15.46:0/999363851 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b8806c360 0x7f8b881102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:01.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.264+0000 7f8b9072a700 1 --2- 172.21.15.46:0/999363851 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b8806b9c0 0x7f8b8806be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:01.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.264+0000 7f8b9072a700 1 -- 172.21.15.46:0/999363851 >> 172.21.15.46:0/999363851 conn(0x7f8b88079290 msgr2=0x7f8b880796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:01.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.264+0000 7f8b9072a700 1 -- 172.21.15.46:0/999363851 shutdown_connections 2024-04-26T21:34:01.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.264+0000 7f8b9072a700 1 -- 172.21.15.46:0/999363851 wait complete. 2024-04-26T21:34:01.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.265+0000 7f8b9072a700 1 Processor -- start 2024-04-26T21:34:01.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.265+0000 7f8b9072a700 1 -- start start 2024-04-26T21:34:01.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.265+0000 7f8b9072a700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b8806b9c0 0x7f8b881023e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:01.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.265+0000 7f8b9072a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b8806c360 0x7f8b88102920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:01.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.265+0000 7f8b9072a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b88109490 0x7f8b88102e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.265+0000 7f8b9072a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8b881135a0 con 0x7f8b8806c360 2024-04-26T21:34:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.265+0000 7f8b9072a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8b88113420 con 0x7f8b88109490 2024-04-26T21:34:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.265+0000 7f8b9072a700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8b88113720 con 0x7f8b8806b9c0 2024-04-26T21:34:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.265+0000 7f8b8dcc5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b8806c360 0x7f8b88102920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.265+0000 7f8b8dcc5700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b8806c360 0x7f8b88102920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41828/0 (socket says 172.21.15.46:41828) 2024-04-26T21:34:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.265+0000 7f8b8dcc5700 1 -- 172.21.15.46:0/2247233556 learned_addr learned my addr 172.21.15.46:0/2247233556 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.265+0000 7f8b8ecc7700 1 --2- 172.21.15.46:0/2247233556 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b88109490 0x7f8b88102e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.266+0000 7f8b8e4c6700 1 --2- 172.21.15.46:0/2247233556 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b8806b9c0 0x7f8b881023e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.266+0000 7f8b8dcc5700 1 -- 172.21.15.46:0/2247233556 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b8806b9c0 msgr2=0x7f8b881023e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.266+0000 7f8b8dcc5700 1 --2- 172.21.15.46:0/2247233556 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b8806b9c0 0x7f8b881023e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.266+0000 7f8b8dcc5700 1 -- 172.21.15.46:0/2247233556 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b88109490 msgr2=0x7f8b88102e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.266+0000 7f8b8dcc5700 1 --2- 172.21.15.46:0/2247233556 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b88109490 0x7f8b88102e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.266+0000 7f8b8dcc5700 1 -- 172.21.15.46:0/2247233556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b881a8020 con 0x7f8b8806c360 2024-04-26T21:34:01.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.266+0000 7f8b8dcc5700 1 --2- 172.21.15.46:0/2247233556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b8806c360 0x7f8b88102920 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8b7800c970 tx=0x7f8b7800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:01.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.266+0000 7f8b7f7fe700 1 -- 172.21.15.46:0/2247233556 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b780078d0 con 0x7f8b8806c360 2024-04-26T21:34:01.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.266+0000 7f8b9072a700 1 -- 172.21.15.46:0/2247233556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b881a8310 con 0x7f8b8806c360 2024-04-26T21:34:01.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.266+0000 7f8b7f7fe700 1 -- 172.21.15.46:0/2247233556 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b78018480 con 0x7f8b8806c360 2024-04-26T21:34:01.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.266+0000 7f8b7f7fe700 1 -- 172.21.15.46:0/2247233556 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7800f6b0 con 0x7f8b8806c360 2024-04-26T21:34:01.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.266+0000 7f8b9072a700 1 -- 172.21.15.46:0/2247233556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b881a8850 con 0x7f8b8806c360 2024-04-26T21:34:01.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.267+0000 7f8b9072a700 1 -- 172.21.15.46:0/2247233556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b8806a110 con 0x7f8b8806c360 2024-04-26T21:34:01.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.269+0000 7f8b7f7fe700 1 -- 172.21.15.46:0/2247233556 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8b78018930 con 0x7f8b8806c360 2024-04-26T21:34:01.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.269+0000 7f8b7f7fe700 1 --2- 172.21.15.46:0/2247233556 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8b740a3cd0 0x7f8b740a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:01.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.270+0000 7f8b8e4c6700 1 --2- 172.21.15.46:0/2247233556 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8b740a3cd0 0x7f8b740a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:01.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.270+0000 7f8b7f7fe700 1 -- 172.21.15.46:0/2247233556 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f8b780d0600 con 0x7f8b8806c360 2024-04-26T21:34:01.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.270+0000 7f8b8e4c6700 1 --2- 172.21.15.46:0/2247233556 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8b740a3cd0 0x7f8b740a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8b8400b5c0 tx=0x7f8b84005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:01.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.274+0000 7f8b7f7fe700 1 -- 172.21.15.46:0/2247233556 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8b7809cf40 con 0x7f8b8806c360 2024-04-26T21:34:01.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:00 smithi142 ceph-mon[32944]: pgmap v332: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:34:01.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:00 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/4235675096' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T21:34:01.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:00 smithi050 ceph-mon[32753]: pgmap v332: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:34:01.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:00 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/4235675096' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T21:34:01.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.493+0000 7f8b9072a700 1 -- 172.21.15.46:0/2247233556 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f8b8802e490 con 0x7f8b8806c360 2024-04-26T21:34:01.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.493+0000 7f8b7f7fe700 1 -- 172.21.15.46:0/2247233556 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f8b7809c840 con 0x7f8b8806c360 2024-04-26T21:34:01.495 INFO:teuthology.orchestra.run.smithi046.stdout:309237645346 2024-04-26T21:34:01.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.495+0000 7f8b9072a700 1 -- 172.21.15.46:0/2247233556 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8b740a3cd0 msgr2=0x7f8b740a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:01.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.495+0000 7f8b9072a700 1 --2- 172.21.15.46:0/2247233556 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8b740a3cd0 0x7f8b740a6190 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8b8400b5c0 tx=0x7f8b84005960 comp rx=0 tx=0).stop 2024-04-26T21:34:01.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.495+0000 7f8b9072a700 1 -- 172.21.15.46:0/2247233556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b8806c360 msgr2=0x7f8b88102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:01.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.495+0000 7f8b9072a700 1 --2- 172.21.15.46:0/2247233556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b8806c360 0x7f8b88102920 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8b7800c970 tx=0x7f8b7800cc80 comp rx=0 tx=0).stop 2024-04-26T21:34:01.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.496+0000 7f8b9072a700 1 -- 172.21.15.46:0/2247233556 shutdown_connections 2024-04-26T21:34:01.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.496+0000 7f8b9072a700 1 --2- 172.21.15.46:0/2247233556 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b88109490 0x7f8b88102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:01.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.496+0000 7f8b9072a700 1 --2- 172.21.15.46:0/2247233556 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b8806b9c0 0x7f8b881023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:01.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.496+0000 7f8b9072a700 1 --2- 172.21.15.46:0/2247233556 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8b740a3cd0 0x7f8b740a6190 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:01.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.496+0000 7f8b9072a700 1 --2- 172.21.15.46:0/2247233556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b8806c360 0x7f8b88102920 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:01.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.496+0000 7f8b9072a700 1 -- 172.21.15.46:0/2247233556 >> 172.21.15.46:0/2247233556 conn(0x7f8b88079290 msgr2=0x7f8b881004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:01.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.496+0000 7f8b9072a700 1 -- 172.21.15.46:0/2247233556 shutdown_connections 2024-04-26T21:34:01.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:01.496+0000 7f8b9072a700 1 -- 172.21.15.46:0/2247233556 wait complete. 2024-04-26T21:34:01.752 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411396 got 77309411402 for osd.2 2024-04-26T21:34:01.752 DEBUG:teuthology.parallel:result is None 2024-04-26T21:34:02.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:01 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2247233556' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T21:34:02.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:01 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/2247233556' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T21:34:02.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:01 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/2247233556' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T21:34:02.374 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645340 got 309237645346 for osd.10 2024-04-26T21:34:02.374 DEBUG:teuthology.parallel:result is None 2024-04-26T21:34:02.374 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-26T21:34:02.375 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T21:34:02.649 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:34:03.844 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:03 smithi046 ceph-mon[29201]: pgmap v333: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:03.844 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:03 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:34:03.844 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:03 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:34:03.844 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:03 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:34:03.844 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:03 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:34:03.844 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:03 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:34:03.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:03 smithi050 ceph-mon[32753]: pgmap v333: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:03.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:03 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:34:03.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:03 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:34:03.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:03 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:34:03.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:03 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:34:03.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:03 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:34:04.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:03 smithi142 ceph-mon[32944]: pgmap v333: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:04.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:03 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:34:04.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:03 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:34:04.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:03 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:34:04.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:03 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:34:04.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:03 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:34:04.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.408+0000 7fb277d41700 1 -- 172.21.15.46:0/3455702329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb270107640 msgr2=0x7fb27010f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:04.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.408+0000 7fb277d41700 1 --2- 172.21.15.46:0/3455702329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb270107640 0x7fb27010f210 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb26800b0d0 tx=0x7fb26800b3e0 comp rx=0 tx=0).stop 2024-04-26T21:34:04.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.408+0000 7fb277d41700 1 -- 172.21.15.46:0/3455702329 shutdown_connections 2024-04-26T21:34:04.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.408+0000 7fb277d41700 1 --2- 172.21.15.46:0/3455702329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb270107640 0x7fb27010f210 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:04.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.408+0000 7fb277d41700 1 --2- 172.21.15.46:0/3455702329 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb2701025b0 0x7fb270102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:04.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.408+0000 7fb277d41700 1 --2- 172.21.15.46:0/3455702329 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb270102f60 0x7fb270106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:04.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.408+0000 7fb277d41700 1 -- 172.21.15.46:0/3455702329 >> 172.21.15.46:0/3455702329 conn(0x7fb2700792d0 msgr2=0x7fb2700796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:04.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.408+0000 7fb277d41700 1 -- 172.21.15.46:0/3455702329 shutdown_connections 2024-04-26T21:34:04.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.408+0000 7fb277d41700 1 -- 172.21.15.46:0/3455702329 wait complete. 2024-04-26T21:34:04.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.409+0000 7fb277d41700 1 Processor -- start 2024-04-26T21:34:04.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.409+0000 7fb277d41700 1 -- start start 2024-04-26T21:34:04.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.409+0000 7fb277d41700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2701025b0 0x7fb270104000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:04.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.409+0000 7fb277d41700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb270102f60 0x7fb270104540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:04.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.409+0000 7fb277d41700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb270107640 0x7fb270104a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:04.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.409+0000 7fb277d41700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb270112550 con 0x7fb270102f60 2024-04-26T21:34:04.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.409+0000 7fb277d41700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb2701123d0 con 0x7fb2701025b0 2024-04-26T21:34:04.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.409+0000 7fb277d41700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb2701126d0 con 0x7fb270107640 2024-04-26T21:34:04.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.409+0000 7fb2752dc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb270102f60 0x7fb270104540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:04.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.410+0000 7fb2752dc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb270102f60 0x7fb270104540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41848/0 (socket says 172.21.15.46:41848) 2024-04-26T21:34:04.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.410+0000 7fb2752dc700 1 -- 172.21.15.46:0/1485522739 learned_addr learned my addr 172.21.15.46:0/1485522739 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:04.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.410+0000 7fb275add700 1 --2- 172.21.15.46:0/1485522739 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2701025b0 0x7fb270104000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:04.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.410+0000 7fb2762de700 1 --2- 172.21.15.46:0/1485522739 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb270107640 0x7fb270104a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:04.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.410+0000 7fb2752dc700 1 -- 172.21.15.46:0/1485522739 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb270107640 msgr2=0x7fb270104a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:04.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.410+0000 7fb2752dc700 1 --2- 172.21.15.46:0/1485522739 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb270107640 0x7fb270104a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:04.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.410+0000 7fb2752dc700 1 -- 172.21.15.46:0/1485522739 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2701025b0 msgr2=0x7fb270104000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:04.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.410+0000 7fb2752dc700 1 --2- 172.21.15.46:0/1485522739 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2701025b0 0x7fb270104000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:04.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.410+0000 7fb2752dc700 1 -- 172.21.15.46:0/1485522739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2701a8070 con 0x7fb270102f60 2024-04-26T21:34:04.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.410+0000 7fb2752dc700 1 --2- 172.21.15.46:0/1485522739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb270102f60 0x7fb270104540 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb26c00d950 tx=0x7fb26c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:04.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.410+0000 7fb266ffd700 1 -- 172.21.15.46:0/1485522739 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26c016070 con 0x7fb270102f60 2024-04-26T21:34:04.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.410+0000 7fb266ffd700 1 -- 172.21.15.46:0/1485522739 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb26c004830 con 0x7fb270102f60 2024-04-26T21:34:04.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.410+0000 7fb277d41700 1 -- 172.21.15.46:0/1485522739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2701a8360 con 0x7fb270102f60 2024-04-26T21:34:04.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.411+0000 7fb266ffd700 1 -- 172.21.15.46:0/1485522739 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26c005520 con 0x7fb270102f60 2024-04-26T21:34:04.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.411+0000 7fb277d41700 1 -- 172.21.15.46:0/1485522739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2701a88f0 con 0x7fb270102f60 2024-04-26T21:34:04.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.412+0000 7fb277d41700 1 -- 172.21.15.46:0/1485522739 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb27006a110 con 0x7fb270102f60 2024-04-26T21:34:04.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.413+0000 7fb266ffd700 1 -- 172.21.15.46:0/1485522739 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb26c004220 con 0x7fb270102f60 2024-04-26T21:34:04.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.413+0000 7fb266ffd700 1 --2- 172.21.15.46:0/1485522739 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb25c0a3d50 0x7fb25c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:04.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.413+0000 7fb275add700 1 --2- 172.21.15.46:0/1485522739 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb25c0a3d50 0x7fb25c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:04.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.413+0000 7fb266ffd700 1 -- 172.21.15.46:0/1485522739 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7fb26c0d01f0 con 0x7fb270102f60 2024-04-26T21:34:04.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.414+0000 7fb275add700 1 --2- 172.21.15.46:0/1485522739 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb25c0a3d50 0x7fb25c0a6210 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb260009a10 tx=0x7fb260009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:04.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.418+0000 7fb266ffd700 1 -- 172.21.15.46:0/1485522739 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb26c01f020 con 0x7fb270102f60 2024-04-26T21:34:04.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.621+0000 7fb277d41700 1 -- 172.21.15.46:0/1485522739 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb270063860 con 0x7fb25c0a3d50 2024-04-26T21:34:04.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.629+0000 7fb266ffd700 1 -- 172.21.15.46:0/1485522739 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258931 (secure 0 0 0) 0x7fb270063860 con 0x7fb25c0a3d50 2024-04-26T21:34:04.634 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:34:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.631+0000 7fb277d41700 1 -- 172.21.15.46:0/1485522739 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb25c0a3d50 msgr2=0x7fb25c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.631+0000 7fb277d41700 1 --2- 172.21.15.46:0/1485522739 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb25c0a3d50 0x7fb25c0a6210 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb260009a10 tx=0x7fb260009190 comp rx=0 tx=0).stop 2024-04-26T21:34:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.631+0000 7fb277d41700 1 -- 172.21.15.46:0/1485522739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb270102f60 msgr2=0x7fb270104540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.631+0000 7fb277d41700 1 --2- 172.21.15.46:0/1485522739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb270102f60 0x7fb270104540 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb26c00d950 tx=0x7fb26c00dc60 comp rx=0 tx=0).stop 2024-04-26T21:34:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.631+0000 7fb277d41700 1 -- 172.21.15.46:0/1485522739 shutdown_connections 2024-04-26T21:34:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.632+0000 7fb277d41700 1 --2- 172.21.15.46:0/1485522739 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb270107640 0x7fb270104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.632+0000 7fb277d41700 1 --2- 172.21.15.46:0/1485522739 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb2701025b0 0x7fb270104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.632+0000 7fb277d41700 1 --2- 172.21.15.46:0/1485522739 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb25c0a3d50 0x7fb25c0a6210 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.632+0000 7fb277d41700 1 --2- 172.21.15.46:0/1485522739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb270102f60 0x7fb270104540 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:04.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.632+0000 7fb277d41700 1 -- 172.21.15.46:0/1485522739 >> 172.21.15.46:0/1485522739 conn(0x7fb2700792d0 msgr2=0x7fb27006be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:04.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.632+0000 7fb277d41700 1 -- 172.21.15.46:0/1485522739 shutdown_connections 2024-04-26T21:34:04.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:04.632+0000 7fb277d41700 1 -- 172.21.15.46:0/1485522739 wait complete. 2024-04-26T21:34:04.637 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-04-26T21:34:05.345 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-04-26T21:34:03.950100+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":644471,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":571,"num_read_kb":388,"num_write":340,"num_write_kb":3037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1180848,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":381540,"kb_used_data":11520,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124560284,"statfs":{"total":1151940427776,"available":1151549730816,"internally_reserved":0,"allocated":11796480,"data_stored":7400037,"data_compressed":112764,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19140,"internal_metadata":378713404},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":30,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003037"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.058915+0000","last_change":"2024-04-26T21:32:44.863556+0000","last_active":"2024-04-26T21:33:31.058915+0000","last_peered":"2024-04-26T21:33:31.058915+0000","last_clean":"2024-04-26T21:33:31.058915+0000","last_became_active":"2024-04-26T21:32:44.863217+0000","last_became_peered":"2024-04-26T21:32:44.863217+0000","last_unstale":"2024-04-26T21:33:31.058915+0000","last_undegraded":"2024-04-26T21:33:31.058915+0000","last_fullsized":"2024-04-26T21:33:31.058915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:36:19.416660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.055122+0000","last_change":"2024-04-26T21:32:44.862340+0000","last_active":"2024-04-26T21:33:31.055122+0000","last_peered":"2024-04-26T21:33:31.055122+0000","last_clean":"2024-04-26T21:33:31.055122+0000","last_became_active":"2024-04-26T21:32:44.861753+0000","last_became_peered":"2024-04-26T21:32:44.861753+0000","last_unstale":"2024-04-26T21:33:31.055122+0000","last_undegraded":"2024-04-26T21:33:31.055122+0000","last_fullsized":"2024-04-26T21:33:31.055122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:56:46.395806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.055114+0000","last_change":"2024-04-26T21:32:44.865678+0000","last_active":"2024-04-26T21:33:31.055114+0000","last_peered":"2024-04-26T21:33:31.055114+0000","last_clean":"2024-04-26T21:33:31.055114+0000","last_became_active":"2024-04-26T21:32:44.865001+0000","last_became_peered":"2024-04-26T21:32:44.865001+0000","last_unstale":"2024-04-26T21:33:31.055114+0000","last_undegraded":"2024-04-26T21:33:31.055114+0000","last_fullsized":"2024-04-26T21:33:31.055114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:29:42.803401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056196+0000","last_change":"2024-04-26T21:32:44.869469+0000","last_active":"2024-04-26T21:33:31.056196+0000","last_peered":"2024-04-26T21:33:31.056196+0000","last_clean":"2024-04-26T21:33:31.056196+0000","last_became_active":"2024-04-26T21:32:44.868443+0000","last_became_peered":"2024-04-26T21:32:44.868443+0000","last_unstale":"2024-04-26T21:33:31.056196+0000","last_undegraded":"2024-04-26T21:33:31.056196+0000","last_fullsized":"2024-04-26T21:33:31.056196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:41:43.252283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056239+0000","last_change":"2024-04-26T21:32:44.855500+0000","last_active":"2024-04-26T21:33:31.056239+0000","last_peered":"2024-04-26T21:33:31.056239+0000","last_clean":"2024-04-26T21:33:31.056239+0000","last_became_active":"2024-04-26T21:32:44.854902+0000","last_became_peered":"2024-04-26T21:32:44.854902+0000","last_unstale":"2024-04-26T21:33:31.056239+0000","last_undegraded":"2024-04-26T21:33:31.056239+0000","last_fullsized":"2024-04-26T21:33:31.056239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:14:02.135590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.406892+0000","last_change":"2024-04-26T21:32:44.864742+0000","last_active":"2024-04-26T21:33:31.406892+0000","last_peered":"2024-04-26T21:33:31.406892+0000","last_clean":"2024-04-26T21:33:31.406892+0000","last_became_active":"2024-04-26T21:32:44.864369+0000","last_became_peered":"2024-04-26T21:32:44.864369+0000","last_unstale":"2024-04-26T21:33:31.406892+0000","last_undegraded":"2024-04-26T21:33:31.406892+0000","last_fullsized":"2024-04-26T21:33:31.406892+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:24:43.757233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060095+0000","last_change":"2024-04-26T21:32:44.863158+0000","last_active":"2024-04-26T21:33:31.060095+0000","last_peered":"2024-04-26T21:33:31.060095+0000","last_clean":"2024-04-26T21:33:31.060095+0000","last_became_active":"2024-04-26T21:32:44.862773+0000","last_became_peered":"2024-04-26T21:32:44.862773+0000","last_unstale":"2024-04-26T21:33:31.060095+0000","last_undegraded":"2024-04-26T21:33:31.060095+0000","last_fullsized":"2024-04-26T21:33:31.060095+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:15:44.907270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.061294+0000","last_change":"2024-04-26T21:32:44.864585+0000","last_active":"2024-04-26T21:33:31.061294+0000","last_peered":"2024-04-26T21:33:31.061294+0000","last_clean":"2024-04-26T21:33:31.061294+0000","last_became_active":"2024-04-26T21:32:44.863572+0000","last_became_peered":"2024-04-26T21:32:44.863572+0000","last_unstale":"2024-04-26T21:33:31.061294+0000","last_undegraded":"2024-04-26T21:33:31.061294+0000","last_fullsized":"2024-04-26T21:33:31.061294+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:32:42.481698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.058597+0000","last_change":"2024-04-26T21:32:44.855339+0000","last_active":"2024-04-26T21:33:31.058597+0000","last_peered":"2024-04-26T21:33:31.058597+0000","last_clean":"2024-04-26T21:33:31.058597+0000","last_became_active":"2024-04-26T21:32:44.853923+0000","last_became_peered":"2024-04-26T21:32:44.853923+0000","last_unstale":"2024-04-26T21:33:31.058597+0000","last_undegraded":"2024-04-26T21:33:31.058597+0000","last_fullsized":"2024-04-26T21:33:31.058597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:08:03.274120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.407313+0000","last_change":"2024-04-26T21:32:44.865564+0000","last_active":"2024-04-26T21:33:31.407313+0000","last_peered":"2024-04-26T21:33:31.407313+0000","last_clean":"2024-04-26T21:33:31.407313+0000","last_became_active":"2024-04-26T21:32:44.865209+0000","last_became_peered":"2024-04-26T21:32:44.865209+0000","last_unstale":"2024-04-26T21:33:31.407313+0000","last_undegraded":"2024-04-26T21:33:31.407313+0000","last_fullsized":"2024-04-26T21:33:31.407313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:51:36.388624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060609+0000","last_change":"2024-04-26T21:32:44.865437+0000","last_active":"2024-04-26T21:33:31.060609+0000","last_peered":"2024-04-26T21:33:31.060609+0000","last_clean":"2024-04-26T21:33:31.060609+0000","last_became_active":"2024-04-26T21:32:44.864662+0000","last_became_peered":"2024-04-26T21:32:44.864662+0000","last_unstale":"2024-04-26T21:33:31.060609+0000","last_undegraded":"2024-04-26T21:33:31.060609+0000","last_fullsized":"2024-04-26T21:33:31.060609+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:23:37.867338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056224+0000","last_change":"2024-04-26T21:32:44.869738+0000","last_active":"2024-04-26T21:33:31.056224+0000","last_peered":"2024-04-26T21:33:31.056224+0000","last_clean":"2024-04-26T21:33:31.056224+0000","last_became_active":"2024-04-26T21:32:44.869133+0000","last_became_peered":"2024-04-26T21:32:44.869133+0000","last_unstale":"2024-04-26T21:33:31.056224+0000","last_undegraded":"2024-04-26T21:33:31.056224+0000","last_fullsized":"2024-04-26T21:33:31.056224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:08:29.233128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056606+0000","last_change":"2024-04-26T21:32:44.856970+0000","last_active":"2024-04-26T21:33:31.056606+0000","last_peered":"2024-04-26T21:33:31.056606+0000","last_clean":"2024-04-26T21:33:31.056606+0000","last_became_active":"2024-04-26T21:32:44.856579+0000","last_became_peered":"2024-04-26T21:32:44.856579+0000","last_unstale":"2024-04-26T21:33:31.056606+0000","last_undegraded":"2024-04-26T21:33:31.056606+0000","last_fullsized":"2024-04-26T21:33:31.056606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:24:17.281094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.054658+0000","last_change":"2024-04-26T21:32:44.859688+0000","last_active":"2024-04-26T21:33:31.054658+0000","last_peered":"2024-04-26T21:33:31.054658+0000","last_clean":"2024-04-26T21:33:31.054658+0000","last_became_active":"2024-04-26T21:32:44.859336+0000","last_became_peered":"2024-04-26T21:32:44.859336+0000","last_unstale":"2024-04-26T21:33:31.054658+0000","last_undegraded":"2024-04-26T21:33:31.054658+0000","last_fullsized":"2024-04-26T21:33:31.054658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:55:46.956334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.054539+0000","last_change":"2024-04-26T21:32:44.860647+0000","last_active":"2024-04-26T21:33:31.054539+0000","last_peered":"2024-04-26T21:33:31.054539+0000","last_clean":"2024-04-26T21:33:31.054539+0000","last_became_active":"2024-04-26T21:32:44.860237+0000","last_became_peered":"2024-04-26T21:32:44.860237+0000","last_unstale":"2024-04-26T21:33:31.054539+0000","last_undegraded":"2024-04-26T21:33:31.054539+0000","last_fullsized":"2024-04-26T21:33:31.054539+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:49:10.042990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.057781+0000","last_change":"2024-04-26T21:32:44.865289+0000","last_active":"2024-04-26T21:33:31.057781+0000","last_peered":"2024-04-26T21:33:31.057781+0000","last_clean":"2024-04-26T21:33:31.057781+0000","last_became_active":"2024-04-26T21:32:44.864951+0000","last_became_peered":"2024-04-26T21:32:44.864951+0000","last_unstale":"2024-04-26T21:33:31.057781+0000","last_undegraded":"2024-04-26T21:33:31.057781+0000","last_fullsized":"2024-04-26T21:33:31.057781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:31:51.998634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.059113+0000","last_change":"2024-04-26T21:32:44.857616+0000","last_active":"2024-04-26T21:33:31.059113+0000","last_peered":"2024-04-26T21:33:31.059113+0000","last_clean":"2024-04-26T21:33:31.059113+0000","last_became_active":"2024-04-26T21:32:44.857259+0000","last_became_peered":"2024-04-26T21:32:44.857259+0000","last_unstale":"2024-04-26T21:33:31.059113+0000","last_undegraded":"2024-04-26T21:33:31.059113+0000","last_fullsized":"2024-04-26T21:33:31.059113+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:59:37.412657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.058307+0000","last_change":"2024-04-26T21:32:44.857177+0000","last_active":"2024-04-26T21:33:31.058307+0000","last_peered":"2024-04-26T21:33:31.058307+0000","last_clean":"2024-04-26T21:33:31.058307+0000","last_became_active":"2024-04-26T21:32:44.856810+0000","last_became_peered":"2024-04-26T21:32:44.856810+0000","last_unstale":"2024-04-26T21:33:31.058307+0000","last_undegraded":"2024-04-26T21:33:31.058307+0000","last_fullsized":"2024-04-26T21:33:31.058307+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:31:40.712896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060561+0000","last_change":"2024-04-26T21:32:44.870498+0000","last_active":"2024-04-26T21:33:31.060561+0000","last_peered":"2024-04-26T21:33:31.060561+0000","last_clean":"2024-04-26T21:33:31.060561+0000","last_became_active":"2024-04-26T21:32:44.870170+0000","last_became_peered":"2024-04-26T21:32:44.870170+0000","last_unstale":"2024-04-26T21:33:31.060561+0000","last_undegraded":"2024-04-26T21:33:31.060561+0000","last_fullsized":"2024-04-26T21:33:31.060561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:27.428027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.058080+0000","last_change":"2024-04-26T21:32:43.845192+0000","last_active":"2024-04-26T21:33:31.058080+0000","last_peered":"2024-04-26T21:33:31.058080+0000","last_clean":"2024-04-26T21:33:31.058080+0000","last_became_active":"2024-04-26T21:32:43.844862+0000","last_became_peered":"2024-04-26T21:32:43.844862+0000","last_unstale":"2024-04-26T21:33:31.058080+0000","last_undegraded":"2024-04-26T21:33:31.058080+0000","last_fullsized":"2024-04-26T21:33:31.058080+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:07:00.060047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.061614+0000","last_change":"2024-04-26T21:32:44.867358+0000","last_active":"2024-04-26T21:33:31.061614+0000","last_peered":"2024-04-26T21:33:31.061614+0000","last_clean":"2024-04-26T21:33:31.061614+0000","last_became_active":"2024-04-26T21:32:44.864833+0000","last_became_peered":"2024-04-26T21:32:44.864833+0000","last_unstale":"2024-04-26T21:33:31.061614+0000","last_undegraded":"2024-04-26T21:33:31.061614+0000","last_fullsized":"2024-04-26T21:33:31.061614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:45:50.285471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.061638+0000","last_change":"2024-04-26T21:32:43.849724+0000","last_active":"2024-04-26T21:33:31.061638+0000","last_peered":"2024-04-26T21:33:31.061638+0000","last_clean":"2024-04-26T21:33:31.061638+0000","last_became_active":"2024-04-26T21:32:43.849040+0000","last_became_peered":"2024-04-26T21:32:43.849040+0000","last_unstale":"2024-04-26T21:33:31.061638+0000","last_undegraded":"2024-04-26T21:33:31.061638+0000","last_fullsized":"2024-04-26T21:33:31.061638+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:48.994093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.055328+0000","last_change":"2024-04-26T21:32:44.863095+0000","last_active":"2024-04-26T21:33:31.055328+0000","last_peered":"2024-04-26T21:33:31.055328+0000","last_clean":"2024-04-26T21:33:31.055328+0000","last_became_active":"2024-04-26T21:32:44.862739+0000","last_became_peered":"2024-04-26T21:32:44.862739+0000","last_unstale":"2024-04-26T21:33:31.055328+0000","last_undegraded":"2024-04-26T21:33:31.055328+0000","last_fullsized":"2024-04-26T21:33:31.055328+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:21:19.927248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.057016+0000","last_change":"2024-04-26T21:32:43.854315+0000","last_active":"2024-04-26T21:33:31.057016+0000","last_peered":"2024-04-26T21:33:31.057016+0000","last_clean":"2024-04-26T21:33:31.057016+0000","last_became_active":"2024-04-26T21:32:43.849472+0000","last_became_peered":"2024-04-26T21:32:43.849472+0000","last_unstale":"2024-04-26T21:33:31.057016+0000","last_undegraded":"2024-04-26T21:33:31.057016+0000","last_fullsized":"2024-04-26T21:33:31.057016+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:42:24.213303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.058526+0000","last_change":"2024-04-26T21:32:44.860100+0000","last_active":"2024-04-26T21:33:31.058526+0000","last_peered":"2024-04-26T21:33:31.058526+0000","last_clean":"2024-04-26T21:33:31.058526+0000","last_became_active":"2024-04-26T21:32:44.858890+0000","last_became_peered":"2024-04-26T21:32:44.858890+0000","last_unstale":"2024-04-26T21:33:31.058526+0000","last_undegraded":"2024-04-26T21:33:31.058526+0000","last_fullsized":"2024-04-26T21:33:31.058526+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:22.749764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056725+0000","last_change":"2024-04-26T21:32:43.840719+0000","last_active":"2024-04-26T21:33:31.056725+0000","last_peered":"2024-04-26T21:33:31.056725+0000","last_clean":"2024-04-26T21:33:31.056725+0000","last_became_active":"2024-04-26T21:32:43.840205+0000","last_became_peered":"2024-04-26T21:32:43.840205+0000","last_unstale":"2024-04-26T21:33:31.056725+0000","last_undegraded":"2024-04-26T21:33:31.056725+0000","last_fullsized":"2024-04-26T21:33:31.056725+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:29:41.025573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.071599+0000","last_change":"2024-04-26T21:33:31.071267+0000","last_active":"2024-04-26T21:33:31.071599+0000","last_peered":"2024-04-26T21:33:31.071599+0000","last_clean":"2024-04-26T21:33:31.071599+0000","last_became_active":"2024-04-26T21:33:31.070994+0000","last_became_peered":"2024-04-26T21:33:31.070994+0000","last_unstale":"2024-04-26T21:33:31.071599+0000","last_undegraded":"2024-04-26T21:33:31.071599+0000","last_fullsized":"2024-04-26T21:33:31.071599+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:38:18.586148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,1],"acting":[9,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"85'6","reported_seq":98,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:57.998869+0000","last_change":"2024-04-26T21:32:43.850841+0000","last_active":"2024-04-26T21:33:57.998869+0000","last_peered":"2024-04-26T21:33:57.998869+0000","last_clean":"2024-04-26T21:33:57.998869+0000","last_became_active":"2024-04-26T21:32:43.850512+0000","last_became_peered":"2024-04-26T21:32:43.850512+0000","last_unstale":"2024-04-26T21:33:57.998869+0000","last_undegraded":"2024-04-26T21:33:57.998869+0000","last_fullsized":"2024-04-26T21:33:57.998869+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:13:51.814145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056232+0000","last_change":"2024-04-26T21:32:44.853438+0000","last_active":"2024-04-26T21:33:31.056232+0000","last_peered":"2024-04-26T21:33:31.056232+0000","last_clean":"2024-04-26T21:33:31.056232+0000","last_became_active":"2024-04-26T21:32:44.853126+0000","last_became_peered":"2024-04-26T21:32:44.853126+0000","last_unstale":"2024-04-26T21:33:31.056232+0000","last_undegraded":"2024-04-26T21:33:31.056232+0000","last_fullsized":"2024-04-26T21:33:31.056232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:43.054626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.406822+0000","last_change":"2024-04-26T21:32:43.839230+0000","last_active":"2024-04-26T21:33:31.406822+0000","last_peered":"2024-04-26T21:33:31.406822+0000","last_clean":"2024-04-26T21:33:31.406822+0000","last_became_active":"2024-04-26T21:32:43.838894+0000","last_became_peered":"2024-04-26T21:32:43.838894+0000","last_unstale":"2024-04-26T21:33:31.406822+0000","last_undegraded":"2024-04-26T21:33:31.406822+0000","last_fullsized":"2024-04-26T21:33:31.406822+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:54:29.718314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.407076+0000","last_change":"2024-04-26T21:32:44.859870+0000","last_active":"2024-04-26T21:33:31.407076+0000","last_peered":"2024-04-26T21:33:31.407076+0000","last_clean":"2024-04-26T21:33:31.407076+0000","last_became_active":"2024-04-26T21:32:44.859474+0000","last_became_peered":"2024-04-26T21:32:44.859474+0000","last_unstale":"2024-04-26T21:33:31.407076+0000","last_undegraded":"2024-04-26T21:33:31.407076+0000","last_fullsized":"2024-04-26T21:33:31.407076+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:13:14.854523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056138+0000","last_change":"2024-04-26T21:32:43.841090+0000","last_active":"2024-04-26T21:33:31.056138+0000","last_peered":"2024-04-26T21:33:31.056138+0000","last_clean":"2024-04-26T21:33:31.056138+0000","last_became_active":"2024-04-26T21:32:43.840723+0000","last_became_peered":"2024-04-26T21:32:43.840723+0000","last_unstale":"2024-04-26T21:33:31.056138+0000","last_undegraded":"2024-04-26T21:33:31.056138+0000","last_fullsized":"2024-04-26T21:33:31.056138+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:36:45.799738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.061390+0000","last_change":"2024-04-26T21:32:44.869439+0000","last_active":"2024-04-26T21:33:31.061390+0000","last_peered":"2024-04-26T21:33:31.061390+0000","last_clean":"2024-04-26T21:33:31.061390+0000","last_became_active":"2024-04-26T21:32:44.868696+0000","last_became_peered":"2024-04-26T21:32:44.868696+0000","last_unstale":"2024-04-26T21:33:31.061390+0000","last_undegraded":"2024-04-26T21:33:31.061390+0000","last_fullsized":"2024-04-26T21:33:31.061390+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:46.400937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.406739+0000","last_change":"2024-04-26T21:32:43.852431+0000","last_active":"2024-04-26T21:33:31.406739+0000","last_peered":"2024-04-26T21:33:31.406739+0000","last_clean":"2024-04-26T21:33:31.406739+0000","last_became_active":"2024-04-26T21:32:43.852143+0000","last_became_peered":"2024-04-26T21:32:43.852143+0000","last_unstale":"2024-04-26T21:33:31.406739+0000","last_undegraded":"2024-04-26T21:33:31.406739+0000","last_fullsized":"2024-04-26T21:33:31.406739+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:35.971582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.057707+0000","last_change":"2024-04-26T21:32:44.861118+0000","last_active":"2024-04-26T21:33:31.057707+0000","last_peered":"2024-04-26T21:33:31.057707+0000","last_clean":"2024-04-26T21:33:31.057707+0000","last_became_active":"2024-04-26T21:32:44.860765+0000","last_became_peered":"2024-04-26T21:32:44.860765+0000","last_unstale":"2024-04-26T21:33:31.057707+0000","last_undegraded":"2024-04-26T21:33:31.057707+0000","last_fullsized":"2024-04-26T21:33:31.057707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:01:09.370401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":92,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:57.998218+0000","last_change":"2024-04-26T21:32:43.840267+0000","last_active":"2024-04-26T21:33:57.998218+0000","last_peered":"2024-04-26T21:33:57.998218+0000","last_clean":"2024-04-26T21:33:57.998218+0000","last_became_active":"2024-04-26T21:32:43.839944+0000","last_became_peered":"2024-04-26T21:32:43.839944+0000","last_unstale":"2024-04-26T21:33:57.998218+0000","last_undegraded":"2024-04-26T21:33:57.998218+0000","last_fullsized":"2024-04-26T21:33:57.998218+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:46:13.671063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.058451+0000","last_change":"2024-04-26T21:32:44.861772+0000","last_active":"2024-04-26T21:33:31.058451+0000","last_peered":"2024-04-26T21:33:31.058451+0000","last_clean":"2024-04-26T21:33:31.058451+0000","last_became_active":"2024-04-26T21:32:44.860376+0000","last_became_peered":"2024-04-26T21:32:44.860376+0000","last_unstale":"2024-04-26T21:33:31.058451+0000","last_undegraded":"2024-04-26T21:33:31.058451+0000","last_fullsized":"2024-04-26T21:33:31.058451+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:07:29.499706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060319+0000","last_change":"2024-04-26T21:32:43.849923+0000","last_active":"2024-04-26T21:33:31.060319+0000","last_peered":"2024-04-26T21:33:31.060319+0000","last_clean":"2024-04-26T21:33:31.060319+0000","last_became_active":"2024-04-26T21:32:43.848193+0000","last_became_peered":"2024-04-26T21:32:43.848193+0000","last_unstale":"2024-04-26T21:33:31.060319+0000","last_undegraded":"2024-04-26T21:33:31.060319+0000","last_fullsized":"2024-04-26T21:33:31.060319+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:06:00.247084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.055308+0000","last_change":"2024-04-26T21:32:44.866684+0000","last_active":"2024-04-26T21:33:31.055308+0000","last_peered":"2024-04-26T21:33:31.055308+0000","last_clean":"2024-04-26T21:33:31.055308+0000","last_became_active":"2024-04-26T21:32:44.866350+0000","last_became_peered":"2024-04-26T21:32:44.866350+0000","last_unstale":"2024-04-26T21:33:31.055308+0000","last_undegraded":"2024-04-26T21:33:31.055308+0000","last_fullsized":"2024-04-26T21:33:31.055308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:21.306500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.407202+0000","last_change":"2024-04-26T21:32:43.841617+0000","last_active":"2024-04-26T21:33:31.407202+0000","last_peered":"2024-04-26T21:33:31.407202+0000","last_clean":"2024-04-26T21:33:31.407202+0000","last_became_active":"2024-04-26T21:32:43.841302+0000","last_became_peered":"2024-04-26T21:32:43.841302+0000","last_unstale":"2024-04-26T21:33:31.407202+0000","last_undegraded":"2024-04-26T21:33:31.407202+0000","last_fullsized":"2024-04-26T21:33:31.407202+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:16:25.409255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.061757+0000","last_change":"2024-04-26T21:33:31.061757+0000","last_active":"2024-04-26T21:33:31.061757+0000","last_peered":"2024-04-26T21:33:31.061757+0000","last_clean":"2024-04-26T21:33:31.061757+0000","last_became_active":"2024-04-26T21:33:31.061300+0000","last_became_peered":"2024-04-26T21:33:31.061300+0000","last_unstale":"2024-04-26T21:33:31.061757+0000","last_undegraded":"2024-04-26T21:33:31.061757+0000","last_fullsized":"2024-04-26T21:33:31.061757+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:14:41.011436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'25","reported_seq":191,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:59.013262+0000","last_change":"2024-04-26T21:32:43.843122+0000","last_active":"2024-04-26T21:33:59.013262+0000","last_peered":"2024-04-26T21:33:59.013262+0000","last_clean":"2024-04-26T21:33:59.013262+0000","last_became_active":"2024-04-26T21:32:43.842751+0000","last_became_peered":"2024-04-26T21:32:43.842751+0000","last_unstale":"2024-04-26T21:33:59.013262+0000","last_undegraded":"2024-04-26T21:33:59.013262+0000","last_fullsized":"2024-04-26T21:33:59.013262+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:16:58.533030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49477,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":147,"num_read_kb":121,"num_write":26,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056599+0000","last_change":"2024-04-26T21:32:44.872829+0000","last_active":"2024-04-26T21:33:31.056599+0000","last_peered":"2024-04-26T21:33:31.056599+0000","last_clean":"2024-04-26T21:33:31.056599+0000","last_became_active":"2024-04-26T21:32:44.872466+0000","last_became_peered":"2024-04-26T21:32:44.872466+0000","last_unstale":"2024-04-26T21:33:31.056599+0000","last_undegraded":"2024-04-26T21:33:31.056599+0000","last_fullsized":"2024-04-26T21:33:31.056599+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:07:27.698789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060409+0000","last_change":"2024-04-26T21:32:43.840277+0000","last_active":"2024-04-26T21:33:31.060409+0000","last_peered":"2024-04-26T21:33:31.060409+0000","last_clean":"2024-04-26T21:33:31.060409+0000","last_became_active":"2024-04-26T21:32:43.839962+0000","last_became_peered":"2024-04-26T21:32:43.839962+0000","last_unstale":"2024-04-26T21:33:31.060409+0000","last_undegraded":"2024-04-26T21:33:31.060409+0000","last_fullsized":"2024-04-26T21:33:31.060409+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:25:54.475216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056357+0000","last_change":"2024-04-26T21:32:44.866465+0000","last_active":"2024-04-26T21:33:31.056357+0000","last_peered":"2024-04-26T21:33:31.056357+0000","last_clean":"2024-04-26T21:33:31.056357+0000","last_became_active":"2024-04-26T21:32:44.866067+0000","last_became_peered":"2024-04-26T21:32:44.866067+0000","last_unstale":"2024-04-26T21:33:31.056357+0000","last_undegraded":"2024-04-26T21:33:31.056357+0000","last_fullsized":"2024-04-26T21:33:31.056357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:47:03.884436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.406841+0000","last_change":"2024-04-26T21:32:43.851535+0000","last_active":"2024-04-26T21:33:31.406841+0000","last_peered":"2024-04-26T21:33:31.406841+0000","last_clean":"2024-04-26T21:33:31.406841+0000","last_became_active":"2024-04-26T21:32:43.851237+0000","last_became_peered":"2024-04-26T21:32:43.851237+0000","last_unstale":"2024-04-26T21:33:31.406841+0000","last_undegraded":"2024-04-26T21:33:31.406841+0000","last_fullsized":"2024-04-26T21:33:31.406841+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:08:40.624027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060685+0000","last_change":"2024-04-26T21:32:44.874686+0000","last_active":"2024-04-26T21:33:31.060685+0000","last_peered":"2024-04-26T21:33:31.060685+0000","last_clean":"2024-04-26T21:33:31.060685+0000","last_became_active":"2024-04-26T21:32:44.874377+0000","last_became_peered":"2024-04-26T21:32:44.874377+0000","last_unstale":"2024-04-26T21:33:31.060685+0000","last_undegraded":"2024-04-26T21:33:31.060685+0000","last_fullsized":"2024-04-26T21:33:31.060685+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:45:33.627663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'4","reported_seq":96,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:57.997933+0000","last_change":"2024-04-26T21:32:43.846522+0000","last_active":"2024-04-26T21:33:57.997933+0000","last_peered":"2024-04-26T21:33:57.997933+0000","last_clean":"2024-04-26T21:33:57.997933+0000","last_became_active":"2024-04-26T21:32:43.846138+0000","last_became_peered":"2024-04-26T21:32:43.846138+0000","last_unstale":"2024-04-26T21:33:57.997933+0000","last_undegraded":"2024-04-26T21:33:57.997933+0000","last_fullsized":"2024-04-26T21:33:57.997933+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:28:40.257244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060034+0000","last_change":"2024-04-26T21:32:44.872228+0000","last_active":"2024-04-26T21:33:31.060034+0000","last_peered":"2024-04-26T21:33:31.060034+0000","last_clean":"2024-04-26T21:33:31.060034+0000","last_became_active":"2024-04-26T21:32:44.871818+0000","last_became_peered":"2024-04-26T21:32:44.871818+0000","last_unstale":"2024-04-26T21:33:31.060034+0000","last_undegraded":"2024-04-26T21:33:31.060034+0000","last_fullsized":"2024-04-26T21:33:31.060034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:52:32.520802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":59,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.073977+0000","last_change":"2024-04-26T21:33:31.073977+0000","last_active":"2024-04-26T21:33:31.073977+0000","last_peered":"2024-04-26T21:33:31.073977+0000","last_clean":"2024-04-26T21:33:31.073977+0000","last_became_active":"2024-04-26T21:33:31.060006+0000","last_became_peered":"2024-04-26T21:33:31.060006+0000","last_unstale":"2024-04-26T21:33:31.073977+0000","last_undegraded":"2024-04-26T21:33:31.073977+0000","last_fullsized":"2024-04-26T21:33:31.073977+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:13:23.795696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.406782+0000","last_change":"2024-04-26T21:32:44.859853+0000","last_active":"2024-04-26T21:33:31.406782+0000","last_peered":"2024-04-26T21:33:31.406782+0000","last_clean":"2024-04-26T21:33:31.406782+0000","last_became_active":"2024-04-26T21:32:44.859521+0000","last_became_peered":"2024-04-26T21:32:44.859521+0000","last_unstale":"2024-04-26T21:33:31.406782+0000","last_undegraded":"2024-04-26T21:33:31.406782+0000","last_fullsized":"2024-04-26T21:33:31.406782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:48:54.955913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056949+0000","last_change":"2024-04-26T21:32:43.840795+0000","last_active":"2024-04-26T21:33:31.056949+0000","last_peered":"2024-04-26T21:33:31.056949+0000","last_clean":"2024-04-26T21:33:31.056949+0000","last_became_active":"2024-04-26T21:32:43.840443+0000","last_became_peered":"2024-04-26T21:32:43.840443+0000","last_unstale":"2024-04-26T21:33:31.056949+0000","last_undegraded":"2024-04-26T21:33:31.056949+0000","last_fullsized":"2024-04-26T21:33:31.056949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:09:48.944085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.058755+0000","last_change":"2024-04-26T21:32:44.862344+0000","last_active":"2024-04-26T21:33:31.058755+0000","last_peered":"2024-04-26T21:33:31.058755+0000","last_clean":"2024-04-26T21:33:31.058755+0000","last_became_active":"2024-04-26T21:32:44.862029+0000","last_became_peered":"2024-04-26T21:32:44.862029+0000","last_unstale":"2024-04-26T21:33:31.058755+0000","last_undegraded":"2024-04-26T21:33:31.058755+0000","last_fullsized":"2024-04-26T21:33:31.058755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:45:48.833753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.058095+0000","last_change":"2024-04-26T21:32:43.843853+0000","last_active":"2024-04-26T21:33:31.058095+0000","last_peered":"2024-04-26T21:33:31.058095+0000","last_clean":"2024-04-26T21:33:31.058095+0000","last_became_active":"2024-04-26T21:32:43.843525+0000","last_became_peered":"2024-04-26T21:32:43.843525+0000","last_unstale":"2024-04-26T21:33:31.058095+0000","last_undegraded":"2024-04-26T21:33:31.058095+0000","last_fullsized":"2024-04-26T21:33:31.058095+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:11:43.222849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.407172+0000","last_change":"2024-04-26T21:32:44.869123+0000","last_active":"2024-04-26T21:33:31.407172+0000","last_peered":"2024-04-26T21:33:31.407172+0000","last_clean":"2024-04-26T21:33:31.407172+0000","last_became_active":"2024-04-26T21:32:44.868573+0000","last_became_peered":"2024-04-26T21:32:44.868573+0000","last_unstale":"2024-04-26T21:33:31.407172+0000","last_undegraded":"2024-04-26T21:33:31.407172+0000","last_fullsized":"2024-04-26T21:33:31.407172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:48:54.447157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060563+0000","last_change":"2024-04-26T21:32:43.850596+0000","last_active":"2024-04-26T21:33:31.060563+0000","last_peered":"2024-04-26T21:33:31.060563+0000","last_clean":"2024-04-26T21:33:31.060563+0000","last_became_active":"2024-04-26T21:32:43.850218+0000","last_became_peered":"2024-04-26T21:32:43.850218+0000","last_unstale":"2024-04-26T21:33:31.060563+0000","last_undegraded":"2024-04-26T21:33:31.060563+0000","last_fullsized":"2024-04-26T21:33:31.060563+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:34.068783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.406954+0000","last_change":"2024-04-26T21:32:44.863450+0000","last_active":"2024-04-26T21:33:31.406954+0000","last_peered":"2024-04-26T21:33:31.406954+0000","last_clean":"2024-04-26T21:33:31.406954+0000","last_became_active":"2024-04-26T21:32:44.863089+0000","last_became_peered":"2024-04-26T21:32:44.863089+0000","last_unstale":"2024-04-26T21:33:31.406954+0000","last_undegraded":"2024-04-26T21:33:31.406954+0000","last_fullsized":"2024-04-26T21:33:31.406954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:04:51.821073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.406892+0000","last_change":"2024-04-26T21:32:44.855575+0000","last_active":"2024-04-26T21:33:31.406892+0000","last_peered":"2024-04-26T21:33:31.406892+0000","last_clean":"2024-04-26T21:33:31.406892+0000","last_became_active":"2024-04-26T21:32:44.855258+0000","last_became_peered":"2024-04-26T21:32:44.855258+0000","last_unstale":"2024-04-26T21:33:31.406892+0000","last_undegraded":"2024-04-26T21:33:31.406892+0000","last_fullsized":"2024-04-26T21:33:31.406892+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:41:07.559095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":93,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:57.997256+0000","last_change":"2024-04-26T21:32:43.850713+0000","last_active":"2024-04-26T21:33:57.997256+0000","last_peered":"2024-04-26T21:33:57.997256+0000","last_clean":"2024-04-26T21:33:57.997256+0000","last_became_active":"2024-04-26T21:32:43.850406+0000","last_became_peered":"2024-04-26T21:32:43.850406+0000","last_unstale":"2024-04-26T21:33:57.997256+0000","last_undegraded":"2024-04-26T21:33:57.997256+0000","last_fullsized":"2024-04-26T21:33:57.997256+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:11:02.682963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.061377+0000","last_change":"2024-04-26T21:32:44.868317+0000","last_active":"2024-04-26T21:33:31.061377+0000","last_peered":"2024-04-26T21:33:31.061377+0000","last_clean":"2024-04-26T21:33:31.061377+0000","last_became_active":"2024-04-26T21:32:44.864571+0000","last_became_peered":"2024-04-26T21:32:44.864571+0000","last_unstale":"2024-04-26T21:33:31.061377+0000","last_undegraded":"2024-04-26T21:33:31.061377+0000","last_fullsized":"2024-04-26T21:33:31.061377+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:24:50.476949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056133+0000","last_change":"2024-04-26T21:32:43.833773+0000","last_active":"2024-04-26T21:33:31.056133+0000","last_peered":"2024-04-26T21:33:31.056133+0000","last_clean":"2024-04-26T21:33:31.056133+0000","last_became_active":"2024-04-26T21:32:43.833338+0000","last_became_peered":"2024-04-26T21:32:43.833338+0000","last_unstale":"2024-04-26T21:33:31.056133+0000","last_undegraded":"2024-04-26T21:33:31.056133+0000","last_fullsized":"2024-04-26T21:33:31.056133+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:15.382494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":340,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060158+0000","last_change":"2024-04-26T21:31:27.930370+0000","last_active":"2024-04-26T21:33:31.060158+0000","last_peered":"2024-04-26T21:33:31.060158+0000","last_clean":"2024-04-26T21:33:31.060158+0000","last_became_active":"2024-04-26T21:31:27.929859+0000","last_became_peered":"2024-04-26T21:31:27.929859+0000","last_unstale":"2024-04-26T21:33:31.060158+0000","last_undegraded":"2024-04-26T21:33:31.060158+0000","last_fullsized":"2024-04-26T21:33:31.060158+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:58.528699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:58.528699+0000","last_clean_scrub_stamp":"2024-04-26T21:27:58.528699+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:00:55.701865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.055991+0000","last_change":"2024-04-26T21:32:44.858891+0000","last_active":"2024-04-26T21:33:31.055991+0000","last_peered":"2024-04-26T21:33:31.055991+0000","last_clean":"2024-04-26T21:33:31.055991+0000","last_became_active":"2024-04-26T21:32:44.858406+0000","last_became_peered":"2024-04-26T21:32:44.858406+0000","last_unstale":"2024-04-26T21:33:31.055991+0000","last_undegraded":"2024-04-26T21:33:31.055991+0000","last_fullsized":"2024-04-26T21:33:31.055991+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:40:13.084352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060332+0000","last_change":"2024-04-26T21:32:43.849442+0000","last_active":"2024-04-26T21:33:31.060332+0000","last_peered":"2024-04-26T21:33:31.060332+0000","last_clean":"2024-04-26T21:33:31.060332+0000","last_became_active":"2024-04-26T21:32:43.847951+0000","last_became_peered":"2024-04-26T21:32:43.847951+0000","last_unstale":"2024-04-26T21:33:31.060332+0000","last_undegraded":"2024-04-26T21:33:31.060332+0000","last_fullsized":"2024-04-26T21:33:31.060332+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:55:00.025931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056679+0000","last_change":"2024-04-26T21:32:43.846736+0000","last_active":"2024-04-26T21:33:31.056679+0000","last_peered":"2024-04-26T21:33:31.056679+0000","last_clean":"2024-04-26T21:33:31.056679+0000","last_became_active":"2024-04-26T21:32:43.846451+0000","last_became_peered":"2024-04-26T21:32:43.846451+0000","last_unstale":"2024-04-26T21:33:31.056679+0000","last_undegraded":"2024-04-26T21:33:31.056679+0000","last_fullsized":"2024-04-26T21:33:31.056679+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:38:46.301597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.057016+0000","last_change":"2024-04-26T21:32:44.856817+0000","last_active":"2024-04-26T21:33:31.057016+0000","last_peered":"2024-04-26T21:33:31.057016+0000","last_clean":"2024-04-26T21:33:31.057016+0000","last_became_active":"2024-04-26T21:32:44.856489+0000","last_became_peered":"2024-04-26T21:32:44.856489+0000","last_unstale":"2024-04-26T21:33:31.057016+0000","last_undegraded":"2024-04-26T21:33:31.057016+0000","last_fullsized":"2024-04-26T21:33:31.057016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:13:33.851351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.058457+0000","last_change":"2024-04-26T21:32:43.838974+0000","last_active":"2024-04-26T21:33:31.058457+0000","last_peered":"2024-04-26T21:33:31.058457+0000","last_clean":"2024-04-26T21:33:31.058457+0000","last_became_active":"2024-04-26T21:32:43.838661+0000","last_became_peered":"2024-04-26T21:32:43.838661+0000","last_unstale":"2024-04-26T21:33:31.058457+0000","last_undegraded":"2024-04-26T21:33:31.058457+0000","last_fullsized":"2024-04-26T21:33:31.058457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:44:29.536559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.061606+0000","last_change":"2024-04-26T21:32:44.872716+0000","last_active":"2024-04-26T21:33:31.061606+0000","last_peered":"2024-04-26T21:33:31.061606+0000","last_clean":"2024-04-26T21:33:31.061606+0000","last_became_active":"2024-04-26T21:32:44.872348+0000","last_became_peered":"2024-04-26T21:32:44.872348+0000","last_unstale":"2024-04-26T21:33:31.061606+0000","last_undegraded":"2024-04-26T21:33:31.061606+0000","last_fullsized":"2024-04-26T21:33:31.061606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:17:56.890488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056380+0000","last_change":"2024-04-26T21:32:43.853138+0000","last_active":"2024-04-26T21:33:31.056380+0000","last_peered":"2024-04-26T21:33:31.056380+0000","last_clean":"2024-04-26T21:33:31.056380+0000","last_became_active":"2024-04-26T21:32:43.852803+0000","last_became_peered":"2024-04-26T21:32:43.852803+0000","last_unstale":"2024-04-26T21:33:31.056380+0000","last_undegraded":"2024-04-26T21:33:31.056380+0000","last_fullsized":"2024-04-26T21:33:31.056380+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:59:04.790442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.055193+0000","last_change":"2024-04-26T21:32:44.867559+0000","last_active":"2024-04-26T21:33:31.055193+0000","last_peered":"2024-04-26T21:33:31.055193+0000","last_clean":"2024-04-26T21:33:31.055193+0000","last_became_active":"2024-04-26T21:32:44.867229+0000","last_became_peered":"2024-04-26T21:32:44.867229+0000","last_unstale":"2024-04-26T21:33:31.055193+0000","last_undegraded":"2024-04-26T21:33:31.055193+0000","last_fullsized":"2024-04-26T21:33:31.055193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:09:09.052753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.057864+0000","last_change":"2024-04-26T21:32:43.844761+0000","last_active":"2024-04-26T21:33:31.057864+0000","last_peered":"2024-04-26T21:33:31.057864+0000","last_clean":"2024-04-26T21:33:31.057864+0000","last_became_active":"2024-04-26T21:32:43.844257+0000","last_became_peered":"2024-04-26T21:32:43.844257+0000","last_unstale":"2024-04-26T21:33:31.057864+0000","last_undegraded":"2024-04-26T21:33:31.057864+0000","last_fullsized":"2024-04-26T21:33:31.057864+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:06:55.560467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056125+0000","last_change":"2024-04-26T21:32:44.856662+0000","last_active":"2024-04-26T21:33:31.056125+0000","last_peered":"2024-04-26T21:33:31.056125+0000","last_clean":"2024-04-26T21:33:31.056125+0000","last_became_active":"2024-04-26T21:32:44.856297+0000","last_became_peered":"2024-04-26T21:32:44.856297+0000","last_unstale":"2024-04-26T21:33:31.056125+0000","last_undegraded":"2024-04-26T21:33:31.056125+0000","last_fullsized":"2024-04-26T21:33:31.056125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:48:55.314959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060803+0000","last_change":"2024-04-26T21:32:43.848710+0000","last_active":"2024-04-26T21:33:31.060803+0000","last_peered":"2024-04-26T21:33:31.060803+0000","last_clean":"2024-04-26T21:33:31.060803+0000","last_became_active":"2024-04-26T21:32:43.848379+0000","last_became_peered":"2024-04-26T21:32:43.848379+0000","last_unstale":"2024-04-26T21:33:31.060803+0000","last_undegraded":"2024-04-26T21:33:31.060803+0000","last_fullsized":"2024-04-26T21:33:31.060803+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:26:44.233330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.058122+0000","last_change":"2024-04-26T21:32:44.858327+0000","last_active":"2024-04-26T21:33:31.058122+0000","last_peered":"2024-04-26T21:33:31.058122+0000","last_clean":"2024-04-26T21:33:31.058122+0000","last_became_active":"2024-04-26T21:32:44.858023+0000","last_became_peered":"2024-04-26T21:32:44.858023+0000","last_unstale":"2024-04-26T21:33:31.058122+0000","last_undegraded":"2024-04-26T21:33:31.058122+0000","last_fullsized":"2024-04-26T21:33:31.058122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:30.588063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.068501+0000","last_change":"2024-04-26T21:33:31.068501+0000","last_active":"2024-04-26T21:33:31.068501+0000","last_peered":"2024-04-26T21:33:31.068501+0000","last_clean":"2024-04-26T21:33:31.068501+0000","last_became_active":"2024-04-26T21:33:31.063005+0000","last_became_peered":"2024-04-26T21:33:31.063005+0000","last_unstale":"2024-04-26T21:33:31.068501+0000","last_undegraded":"2024-04-26T21:33:31.068501+0000","last_fullsized":"2024-04-26T21:33:31.068501+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:50:04.843538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.061020+0000","last_change":"2024-04-26T21:32:44.867088+0000","last_active":"2024-04-26T21:33:31.061020+0000","last_peered":"2024-04-26T21:33:31.061020+0000","last_clean":"2024-04-26T21:33:31.061020+0000","last_became_active":"2024-04-26T21:32:44.863623+0000","last_became_peered":"2024-04-26T21:32:44.863623+0000","last_unstale":"2024-04-26T21:33:31.061020+0000","last_undegraded":"2024-04-26T21:33:31.061020+0000","last_fullsized":"2024-04-26T21:33:31.061020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:42:21.107362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.066518+0000","last_change":"2024-04-26T21:33:31.066518+0000","last_active":"2024-04-26T21:33:31.066518+0000","last_peered":"2024-04-26T21:33:31.066518+0000","last_clean":"2024-04-26T21:33:31.066518+0000","last_became_active":"2024-04-26T21:33:31.061769+0000","last_became_peered":"2024-04-26T21:33:31.061769+0000","last_unstale":"2024-04-26T21:33:31.066518+0000","last_undegraded":"2024-04-26T21:33:31.066518+0000","last_fullsized":"2024-04-26T21:33:31.066518+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:56:30.724712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060839+0000","last_change":"2024-04-26T21:32:44.863365+0000","last_active":"2024-04-26T21:33:31.060839+0000","last_peered":"2024-04-26T21:33:31.060839+0000","last_clean":"2024-04-26T21:33:31.060839+0000","last_became_active":"2024-04-26T21:32:44.863012+0000","last_became_peered":"2024-04-26T21:32:44.863012+0000","last_unstale":"2024-04-26T21:33:31.060839+0000","last_undegraded":"2024-04-26T21:33:31.060839+0000","last_fullsized":"2024-04-26T21:33:31.060839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:09:50.023773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060732+0000","last_change":"2024-04-26T21:32:43.853507+0000","last_active":"2024-04-26T21:33:31.060732+0000","last_peered":"2024-04-26T21:33:31.060732+0000","last_clean":"2024-04-26T21:33:31.060732+0000","last_became_active":"2024-04-26T21:32:43.852680+0000","last_became_peered":"2024-04-26T21:32:43.852680+0000","last_unstale":"2024-04-26T21:33:31.060732+0000","last_undegraded":"2024-04-26T21:33:31.060732+0000","last_fullsized":"2024-04-26T21:33:31.060732+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:46:51.836736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060622+0000","last_change":"2024-04-26T21:32:44.865158+0000","last_active":"2024-04-26T21:33:31.060622+0000","last_peered":"2024-04-26T21:33:31.060622+0000","last_clean":"2024-04-26T21:33:31.060622+0000","last_became_active":"2024-04-26T21:32:44.864182+0000","last_became_peered":"2024-04-26T21:32:44.864182+0000","last_unstale":"2024-04-26T21:33:31.060622+0000","last_undegraded":"2024-04-26T21:33:31.060622+0000","last_fullsized":"2024-04-26T21:33:31.060622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:55.918623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.407171+0000","last_change":"2024-04-26T21:32:43.842913+0000","last_active":"2024-04-26T21:33:31.407171+0000","last_peered":"2024-04-26T21:33:31.407171+0000","last_clean":"2024-04-26T21:33:31.407171+0000","last_became_active":"2024-04-26T21:32:43.842599+0000","last_became_peered":"2024-04-26T21:32:43.842599+0000","last_unstale":"2024-04-26T21:33:31.407171+0000","last_undegraded":"2024-04-26T21:33:31.407171+0000","last_fullsized":"2024-04-26T21:33:31.407171+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:46.583398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060236+0000","last_change":"2024-04-26T21:32:44.864239+0000","last_active":"2024-04-26T21:33:31.060236+0000","last_peered":"2024-04-26T21:33:31.060236+0000","last_clean":"2024-04-26T21:33:31.060236+0000","last_became_active":"2024-04-26T21:32:44.863862+0000","last_became_peered":"2024-04-26T21:32:44.863862+0000","last_unstale":"2024-04-26T21:33:31.060236+0000","last_undegraded":"2024-04-26T21:33:31.060236+0000","last_fullsized":"2024-04-26T21:33:31.060236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:32:16.958044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056742+0000","last_change":"2024-04-26T21:32:43.849768+0000","last_active":"2024-04-26T21:33:31.056742+0000","last_peered":"2024-04-26T21:33:31.056742+0000","last_clean":"2024-04-26T21:33:31.056742+0000","last_became_active":"2024-04-26T21:32:43.849424+0000","last_became_peered":"2024-04-26T21:32:43.849424+0000","last_unstale":"2024-04-26T21:33:31.056742+0000","last_undegraded":"2024-04-26T21:33:31.056742+0000","last_fullsized":"2024-04-26T21:33:31.056742+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:46:23.231819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060448+0000","last_change":"2024-04-26T21:32:44.862597+0000","last_active":"2024-04-26T21:33:31.060448+0000","last_peered":"2024-04-26T21:33:31.060448+0000","last_clean":"2024-04-26T21:33:31.060448+0000","last_became_active":"2024-04-26T21:32:44.862259+0000","last_became_peered":"2024-04-26T21:32:44.862259+0000","last_unstale":"2024-04-26T21:33:31.060448+0000","last_undegraded":"2024-04-26T21:33:31.060448+0000","last_fullsized":"2024-04-26T21:33:31.060448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:19.759329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.058228+0000","last_change":"2024-04-26T21:32:44.855196+0000","last_active":"2024-04-26T21:33:31.058228+0000","last_peered":"2024-04-26T21:33:31.058228+0000","last_clean":"2024-04-26T21:33:31.058228+0000","last_became_active":"2024-04-26T21:32:44.854774+0000","last_became_peered":"2024-04-26T21:32:44.854774+0000","last_unstale":"2024-04-26T21:33:31.058228+0000","last_undegraded":"2024-04-26T21:33:31.058228+0000","last_fullsized":"2024-04-26T21:33:31.058228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:46:18.370983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060704+0000","last_change":"2024-04-26T21:32:44.868115+0000","last_active":"2024-04-26T21:33:31.060704+0000","last_peered":"2024-04-26T21:33:31.060704+0000","last_clean":"2024-04-26T21:33:31.060704+0000","last_became_active":"2024-04-26T21:32:44.867767+0000","last_became_peered":"2024-04-26T21:32:44.867767+0000","last_unstale":"2024-04-26T21:33:31.060704+0000","last_undegraded":"2024-04-26T21:33:31.060704+0000","last_fullsized":"2024-04-26T21:33:31.060704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:30:22.130786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060399+0000","last_change":"2024-04-26T21:32:44.864633+0000","last_active":"2024-04-26T21:33:31.060399+0000","last_peered":"2024-04-26T21:33:31.060399+0000","last_clean":"2024-04-26T21:33:31.060399+0000","last_became_active":"2024-04-26T21:32:44.864245+0000","last_became_peered":"2024-04-26T21:32:44.864245+0000","last_unstale":"2024-04-26T21:33:31.060399+0000","last_undegraded":"2024-04-26T21:33:31.060399+0000","last_fullsized":"2024-04-26T21:33:31.060399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:10:20.315396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.054775+0000","last_change":"2024-04-26T21:32:44.858122+0000","last_active":"2024-04-26T21:33:31.054775+0000","last_peered":"2024-04-26T21:33:31.054775+0000","last_clean":"2024-04-26T21:33:31.054775+0000","last_became_active":"2024-04-26T21:32:44.857513+0000","last_became_peered":"2024-04-26T21:32:44.857513+0000","last_unstale":"2024-04-26T21:33:31.054775+0000","last_undegraded":"2024-04-26T21:33:31.054775+0000","last_fullsized":"2024-04-26T21:33:31.054775+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:50:40.725215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.059869+0000","last_change":"2024-04-26T21:33:31.059869+0000","last_active":"2024-04-26T21:33:31.059869+0000","last_peered":"2024-04-26T21:33:31.059869+0000","last_clean":"2024-04-26T21:33:31.059869+0000","last_became_active":"2024-04-26T21:33:31.059581+0000","last_became_peered":"2024-04-26T21:33:31.059581+0000","last_unstale":"2024-04-26T21:33:31.059869+0000","last_undegraded":"2024-04-26T21:33:31.059869+0000","last_fullsized":"2024-04-26T21:33:31.059869+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:05:21.289959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.059005+0000","last_change":"2024-04-26T21:32:44.852073+0000","last_active":"2024-04-26T21:33:31.059005+0000","last_peered":"2024-04-26T21:33:31.059005+0000","last_clean":"2024-04-26T21:33:31.059005+0000","last_became_active":"2024-04-26T21:32:44.851717+0000","last_became_peered":"2024-04-26T21:32:44.851717+0000","last_unstale":"2024-04-26T21:33:31.059005+0000","last_undegraded":"2024-04-26T21:33:31.059005+0000","last_fullsized":"2024-04-26T21:33:31.059005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:27:37.599933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060852+0000","last_change":"2024-04-26T21:32:44.859503+0000","last_active":"2024-04-26T21:33:31.060852+0000","last_peered":"2024-04-26T21:33:31.060852+0000","last_clean":"2024-04-26T21:33:31.060852+0000","last_became_active":"2024-04-26T21:32:44.859009+0000","last_became_peered":"2024-04-26T21:32:44.859009+0000","last_unstale":"2024-04-26T21:33:31.060852+0000","last_undegraded":"2024-04-26T21:33:31.060852+0000","last_fullsized":"2024-04-26T21:33:31.060852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:00:26.307567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.062113+0000","last_change":"2024-04-26T21:33:31.061589+0000","last_active":"2024-04-26T21:33:31.062113+0000","last_peered":"2024-04-26T21:33:31.062113+0000","last_clean":"2024-04-26T21:33:31.062113+0000","last_became_active":"2024-04-26T21:33:31.061277+0000","last_became_peered":"2024-04-26T21:33:31.061277+0000","last_unstale":"2024-04-26T21:33:31.062113+0000","last_undegraded":"2024-04-26T21:33:31.062113+0000","last_fullsized":"2024-04-26T21:33:31.062113+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:36:40.775558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.406820+0000","last_change":"2024-04-26T21:32:44.857666+0000","last_active":"2024-04-26T21:33:31.406820+0000","last_peered":"2024-04-26T21:33:31.406820+0000","last_clean":"2024-04-26T21:33:31.406820+0000","last_became_active":"2024-04-26T21:32:44.857342+0000","last_became_peered":"2024-04-26T21:32:44.857342+0000","last_unstale":"2024-04-26T21:33:31.406820+0000","last_undegraded":"2024-04-26T21:33:31.406820+0000","last_fullsized":"2024-04-26T21:33:31.406820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:05:15.139919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060911+0000","last_change":"2024-04-26T21:32:44.868571+0000","last_active":"2024-04-26T21:33:31.060911+0000","last_peered":"2024-04-26T21:33:31.060911+0000","last_clean":"2024-04-26T21:33:31.060911+0000","last_became_active":"2024-04-26T21:32:44.868191+0000","last_became_peered":"2024-04-26T21:32:44.868191+0000","last_unstale":"2024-04-26T21:33:31.060911+0000","last_undegraded":"2024-04-26T21:33:31.060911+0000","last_fullsized":"2024-04-26T21:33:31.060911+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:35.329277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.058384+0000","last_change":"2024-04-26T21:32:44.867823+0000","last_active":"2024-04-26T21:33:31.058384+0000","last_peered":"2024-04-26T21:33:31.058384+0000","last_clean":"2024-04-26T21:33:31.058384+0000","last_became_active":"2024-04-26T21:32:44.867496+0000","last_became_peered":"2024-04-26T21:32:44.867496+0000","last_unstale":"2024-04-26T21:33:31.058384+0000","last_undegraded":"2024-04-26T21:33:31.058384+0000","last_fullsized":"2024-04-26T21:33:31.058384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:41:47.484332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056623+0000","last_change":"2024-04-26T21:32:44.864720+0000","last_active":"2024-04-26T21:33:31.056623+0000","last_peered":"2024-04-26T21:33:31.056623+0000","last_clean":"2024-04-26T21:33:31.056623+0000","last_became_active":"2024-04-26T21:32:44.864395+0000","last_became_peered":"2024-04-26T21:32:44.864395+0000","last_unstale":"2024-04-26T21:33:31.056623+0000","last_undegraded":"2024-04-26T21:33:31.056623+0000","last_fullsized":"2024-04-26T21:33:31.056623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:25.416249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.058608+0000","last_change":"2024-04-26T21:32:44.862415+0000","last_active":"2024-04-26T21:33:31.058608+0000","last_peered":"2024-04-26T21:33:31.058608+0000","last_clean":"2024-04-26T21:33:31.058608+0000","last_became_active":"2024-04-26T21:32:44.862141+0000","last_became_peered":"2024-04-26T21:32:44.862141+0000","last_unstale":"2024-04-26T21:33:31.058608+0000","last_undegraded":"2024-04-26T21:33:31.058608+0000","last_fullsized":"2024-04-26T21:33:31.058608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:52:55.919407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280902,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":53869,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":445,"num_read_kb":279,"num_write":157,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":368640,"data_stored":161607,"data_compressed":5583,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383710,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":456134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645346,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1077248,"data_stored":1097483,"data_compressed":37588,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808872,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":958464,"data_stored":456370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070509,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":958464,"data_stored":456290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332146,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1007616,"data_stored":1048611,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495671,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":946176,"data_stored":455759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574929,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40472,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704680,"statfs":{"total":95995035648,"available":95953592320,"internally_reserved":0,"allocated":991232,"data_stored":505029,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738454,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":457030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411403,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":506658,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182471,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36284,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708868,"statfs":{"total":95995035648,"available":95957880832,"internally_reserved":0,"allocated":962560,"data_stored":456070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920834,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":958464,"data_stored":456699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757309,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1019904,"data_stored":1047904,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34089,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34089,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34089,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":49477,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":51106,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":51446,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T21:34:05.350 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T21:34:05.599 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:34:05.858 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:05 smithi046 ceph-mon[29201]: pgmap v334: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:34:05.858 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:05 smithi046 ceph-mon[29201]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:34:05.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:05 smithi050 ceph-mon[32753]: pgmap v334: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:34:05.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:05 smithi050 ceph-mon[32753]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:34:06.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:05 smithi142 ceph-mon[32944]: pgmap v334: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:34:06.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:05 smithi142 ceph-mon[32944]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:34:07.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.305+0000 7fb700e96700 1 -- 172.21.15.46:0/2730955322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc103e60 msgr2=0x7fb6fc1102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:07.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.305+0000 7fb700e96700 1 --2- 172.21.15.46:0/2730955322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc103e60 0x7fb6fc1102b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb6f400b0a0 tx=0x7fb6f400b3b0 comp rx=0 tx=0).stop 2024-04-26T21:34:07.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.305+0000 7fb700e96700 1 -- 172.21.15.46:0/2730955322 shutdown_connections 2024-04-26T21:34:07.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.305+0000 7fb700e96700 1 --2- 172.21.15.46:0/2730955322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc103e60 0x7fb6fc1102b0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:07.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.305+0000 7fb700e96700 1 --2- 172.21.15.46:0/2730955322 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6fc1094e0 0x7fb6fc1098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:07.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.305+0000 7fb700e96700 1 --2- 172.21.15.46:0/2730955322 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6fc1034a0 0x7fb6fc103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:07.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.305+0000 7fb700e96700 1 -- 172.21.15.46:0/2730955322 >> 172.21.15.46:0/2730955322 conn(0x7fb6fc07a590 msgr2=0x7fb6fc07a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:07.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.305+0000 7fb700e96700 1 -- 172.21.15.46:0/2730955322 shutdown_connections 2024-04-26T21:34:07.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.305+0000 7fb700e96700 1 -- 172.21.15.46:0/2730955322 wait complete. 2024-04-26T21:34:07.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.306+0000 7fb700e96700 1 Processor -- start 2024-04-26T21:34:07.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.306+0000 7fb700e96700 1 -- start start 2024-04-26T21:34:07.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.306+0000 7fb700e96700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc1034a0 0x7fb6fc19f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:07.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.307+0000 7fb700e96700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6fc103e60 0x7fb6fc19fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:07.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.307+0000 7fb700e96700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6fc1094e0 0x7fb6fc199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:07.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.307+0000 7fb700e96700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fb6fc113610 con 0x7fb6fc1034a0 2024-04-26T21:34:07.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.307+0000 7fb700e96700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb6fc113490 con 0x7fb6fc103e60 2024-04-26T21:34:07.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.307+0000 7fb700e96700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fb6fc113790 con 0x7fb6fc1094e0 2024-04-26T21:34:07.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.307+0000 7fb6fa59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc1034a0 0x7fb6fc19f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:07.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.307+0000 7fb6fa59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc1034a0 0x7fb6fc19f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40150/0 (socket says 172.21.15.46:40150) 2024-04-26T21:34:07.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.307+0000 7fb6fa59c700 1 -- 172.21.15.46:0/1387211909 learned_addr learned my addr 172.21.15.46:0/1387211909 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:07.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.307+0000 7fb6f9d9b700 1 --2- 172.21.15.46:0/1387211909 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6fc103e60 0x7fb6fc19fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:07.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.307+0000 7fb6fad9d700 1 --2- 172.21.15.46:0/1387211909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6fc1094e0 0x7fb6fc199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:07.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.307+0000 7fb6fa59c700 1 -- 172.21.15.46:0/1387211909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6fc1094e0 msgr2=0x7fb6fc199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:07.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.307+0000 7fb6fa59c700 1 --2- 172.21.15.46:0/1387211909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6fc1094e0 0x7fb6fc199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:07.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.307+0000 7fb6fa59c700 1 -- 172.21.15.46:0/1387211909 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6fc103e60 msgr2=0x7fb6fc19fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.307+0000 7fb6fa59c700 1 --2- 172.21.15.46:0/1387211909 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6fc103e60 0x7fb6fc19fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.307+0000 7fb6fa59c700 1 -- 172.21.15.46:0/1387211909 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6fc19a090 con 0x7fb6fc1034a0 2024-04-26T21:34:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.307+0000 7fb6fa59c700 1 --2- 172.21.15.46:0/1387211909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc1034a0 0x7fb6fc19f770 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec00e9a0 tx=0x7fb6ec00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.307+0000 7fb6eb7fe700 1 -- 172.21.15.46:0/1387211909 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec00cbd0 con 0x7fb6fc1034a0 2024-04-26T21:34:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.307+0000 7fb6eb7fe700 1 -- 172.21.15.46:0/1387211909 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6ec00cd70 con 0x7fb6fc1034a0 2024-04-26T21:34:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.307+0000 7fb700e96700 1 -- 172.21.15.46:0/1387211909 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6fc19a380 con 0x7fb6fc1034a0 2024-04-26T21:34:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.308+0000 7fb6eb7fe700 1 -- 172.21.15.46:0/1387211909 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec018680 con 0x7fb6fc1034a0 2024-04-26T21:34:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.308+0000 7fb700e96700 1 -- 172.21.15.46:0/1387211909 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6fc1a6640 con 0x7fb6fc1034a0 2024-04-26T21:34:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.309+0000 7fb700e96700 1 -- 172.21.15.46:0/1387211909 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6fc10b950 con 0x7fb6fc1034a0 2024-04-26T21:34:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.310+0000 7fb6eb7fe700 1 -- 172.21.15.46:0/1387211909 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fb6ec010430 con 0x7fb6fc1034a0 2024-04-26T21:34:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.310+0000 7fb6eb7fe700 1 --2- 172.21.15.46:0/1387211909 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb6e40a3d50 0x7fb6e40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.311+0000 7fb6f9d9b700 1 --2- 172.21.15.46:0/1387211909 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb6e40a3d50 0x7fb6e40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.311+0000 7fb6eb7fe700 1 -- 172.21.15.46:0/1387211909 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7fb6ec0d0690 con 0x7fb6fc1034a0 2024-04-26T21:34:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.311+0000 7fb6f9d9b700 1 --2- 172.21.15.46:0/1387211909 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb6e40a3d50 0x7fb6e40a6210 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc07ac20 tx=0x7fb6f0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:07.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.315+0000 7fb6eb7fe700 1 -- 172.21.15.46:0/1387211909 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb6ec09cf40 con 0x7fb6fc1034a0 2024-04-26T21:34:07.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.518+0000 7fb700e96700 1 -- 172.21.15.46:0/1387211909 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb6fc063860 con 0x7fb6e40a3d50 2024-04-26T21:34:07.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.526+0000 7fb6eb7fe700 1 -- 172.21.15.46:0/1387211909 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258933 (secure 0 0 0) 0x7fb6fc063860 con 0x7fb6e40a3d50 2024-04-26T21:34:07.530 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:34:07.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.529+0000 7fb700e96700 1 -- 172.21.15.46:0/1387211909 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb6e40a3d50 msgr2=0x7fb6e40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:07.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.529+0000 7fb700e96700 1 --2- 172.21.15.46:0/1387211909 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb6e40a3d50 0x7fb6e40a6210 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc07ac20 tx=0x7fb6f0009260 comp rx=0 tx=0).stop 2024-04-26T21:34:07.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.529+0000 7fb700e96700 1 -- 172.21.15.46:0/1387211909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc1034a0 msgr2=0x7fb6fc19f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:07.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.529+0000 7fb700e96700 1 --2- 172.21.15.46:0/1387211909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc1034a0 0x7fb6fc19f770 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec00e9a0 tx=0x7fb6ec00ecb0 comp rx=0 tx=0).stop 2024-04-26T21:34:07.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.529+0000 7fb700e96700 1 -- 172.21.15.46:0/1387211909 shutdown_connections 2024-04-26T21:34:07.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.529+0000 7fb700e96700 1 --2- 172.21.15.46:0/1387211909 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fb6e40a3d50 0x7fb6e40a6210 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:07.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.529+0000 7fb700e96700 1 --2- 172.21.15.46:0/1387211909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc1034a0 0x7fb6fc19f770 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:07.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.529+0000 7fb700e96700 1 --2- 172.21.15.46:0/1387211909 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fb6fc1094e0 0x7fb6fc199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:07.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.529+0000 7fb700e96700 1 --2- 172.21.15.46:0/1387211909 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb6fc103e60 0x7fb6fc19fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:07.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.529+0000 7fb700e96700 1 -- 172.21.15.46:0/1387211909 >> 172.21.15.46:0/1387211909 conn(0x7fb6fc07a590 msgr2=0x7fb6fc1021a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:07.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.529+0000 7fb700e96700 1 -- 172.21.15.46:0/1387211909 shutdown_connections 2024-04-26T21:34:07.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:07.529+0000 7fb700e96700 1 -- 172.21.15.46:0/1387211909 wait complete. 2024-04-26T21:34:07.533 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-04-26T21:34:07.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:07 smithi050 ceph-mon[32753]: pgmap v335: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T21:34:07.894 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:07 smithi046 ceph-mon[29201]: pgmap v335: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T21:34:08.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:07 smithi142 ceph-mon[32944]: pgmap v335: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T21:34:08.255 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2024-04-26T21:34:05.950547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":644471,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":601,"num_read_kb":403,"num_write":340,"num_write_kb":3037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1180848,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":381540,"kb_used_data":11520,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124560284,"statfs":{"total":1151940427776,"available":1151549730816,"internally_reserved":0,"allocated":11796480,"data_stored":7400037,"data_compressed":112764,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19140,"internal_metadata":378713404},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":90,"num_read_kb":45,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003039"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.058915+0000","last_change":"2024-04-26T21:32:44.863556+0000","last_active":"2024-04-26T21:33:31.058915+0000","last_peered":"2024-04-26T21:33:31.058915+0000","last_clean":"2024-04-26T21:33:31.058915+0000","last_became_active":"2024-04-26T21:32:44.863217+0000","last_became_peered":"2024-04-26T21:32:44.863217+0000","last_unstale":"2024-04-26T21:33:31.058915+0000","last_undegraded":"2024-04-26T21:33:31.058915+0000","last_fullsized":"2024-04-26T21:33:31.058915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:36:19.416660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.055122+0000","last_change":"2024-04-26T21:32:44.862340+0000","last_active":"2024-04-26T21:33:31.055122+0000","last_peered":"2024-04-26T21:33:31.055122+0000","last_clean":"2024-04-26T21:33:31.055122+0000","last_became_active":"2024-04-26T21:32:44.861753+0000","last_became_peered":"2024-04-26T21:32:44.861753+0000","last_unstale":"2024-04-26T21:33:31.055122+0000","last_undegraded":"2024-04-26T21:33:31.055122+0000","last_fullsized":"2024-04-26T21:33:31.055122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:56:46.395806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.055114+0000","last_change":"2024-04-26T21:32:44.865678+0000","last_active":"2024-04-26T21:33:31.055114+0000","last_peered":"2024-04-26T21:33:31.055114+0000","last_clean":"2024-04-26T21:33:31.055114+0000","last_became_active":"2024-04-26T21:32:44.865001+0000","last_became_peered":"2024-04-26T21:32:44.865001+0000","last_unstale":"2024-04-26T21:33:31.055114+0000","last_undegraded":"2024-04-26T21:33:31.055114+0000","last_fullsized":"2024-04-26T21:33:31.055114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:29:42.803401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056196+0000","last_change":"2024-04-26T21:32:44.869469+0000","last_active":"2024-04-26T21:33:31.056196+0000","last_peered":"2024-04-26T21:33:31.056196+0000","last_clean":"2024-04-26T21:33:31.056196+0000","last_became_active":"2024-04-26T21:32:44.868443+0000","last_became_peered":"2024-04-26T21:32:44.868443+0000","last_unstale":"2024-04-26T21:33:31.056196+0000","last_undegraded":"2024-04-26T21:33:31.056196+0000","last_fullsized":"2024-04-26T21:33:31.056196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:41:43.252283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056239+0000","last_change":"2024-04-26T21:32:44.855500+0000","last_active":"2024-04-26T21:33:31.056239+0000","last_peered":"2024-04-26T21:33:31.056239+0000","last_clean":"2024-04-26T21:33:31.056239+0000","last_became_active":"2024-04-26T21:32:44.854902+0000","last_became_peered":"2024-04-26T21:32:44.854902+0000","last_unstale":"2024-04-26T21:33:31.056239+0000","last_undegraded":"2024-04-26T21:33:31.056239+0000","last_fullsized":"2024-04-26T21:33:31.056239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:14:02.135590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.406892+0000","last_change":"2024-04-26T21:32:44.864742+0000","last_active":"2024-04-26T21:33:31.406892+0000","last_peered":"2024-04-26T21:33:31.406892+0000","last_clean":"2024-04-26T21:33:31.406892+0000","last_became_active":"2024-04-26T21:32:44.864369+0000","last_became_peered":"2024-04-26T21:32:44.864369+0000","last_unstale":"2024-04-26T21:33:31.406892+0000","last_undegraded":"2024-04-26T21:33:31.406892+0000","last_fullsized":"2024-04-26T21:33:31.406892+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:24:43.757233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060095+0000","last_change":"2024-04-26T21:32:44.863158+0000","last_active":"2024-04-26T21:33:31.060095+0000","last_peered":"2024-04-26T21:33:31.060095+0000","last_clean":"2024-04-26T21:33:31.060095+0000","last_became_active":"2024-04-26T21:32:44.862773+0000","last_became_peered":"2024-04-26T21:32:44.862773+0000","last_unstale":"2024-04-26T21:33:31.060095+0000","last_undegraded":"2024-04-26T21:33:31.060095+0000","last_fullsized":"2024-04-26T21:33:31.060095+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:15:44.907270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.061294+0000","last_change":"2024-04-26T21:32:44.864585+0000","last_active":"2024-04-26T21:33:31.061294+0000","last_peered":"2024-04-26T21:33:31.061294+0000","last_clean":"2024-04-26T21:33:31.061294+0000","last_became_active":"2024-04-26T21:32:44.863572+0000","last_became_peered":"2024-04-26T21:32:44.863572+0000","last_unstale":"2024-04-26T21:33:31.061294+0000","last_undegraded":"2024-04-26T21:33:31.061294+0000","last_fullsized":"2024-04-26T21:33:31.061294+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:32:42.481698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.058597+0000","last_change":"2024-04-26T21:32:44.855339+0000","last_active":"2024-04-26T21:33:31.058597+0000","last_peered":"2024-04-26T21:33:31.058597+0000","last_clean":"2024-04-26T21:33:31.058597+0000","last_became_active":"2024-04-26T21:32:44.853923+0000","last_became_peered":"2024-04-26T21:32:44.853923+0000","last_unstale":"2024-04-26T21:33:31.058597+0000","last_undegraded":"2024-04-26T21:33:31.058597+0000","last_fullsized":"2024-04-26T21:33:31.058597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:08:03.274120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.407313+0000","last_change":"2024-04-26T21:32:44.865564+0000","last_active":"2024-04-26T21:33:31.407313+0000","last_peered":"2024-04-26T21:33:31.407313+0000","last_clean":"2024-04-26T21:33:31.407313+0000","last_became_active":"2024-04-26T21:32:44.865209+0000","last_became_peered":"2024-04-26T21:32:44.865209+0000","last_unstale":"2024-04-26T21:33:31.407313+0000","last_undegraded":"2024-04-26T21:33:31.407313+0000","last_fullsized":"2024-04-26T21:33:31.407313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:51:36.388624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060609+0000","last_change":"2024-04-26T21:32:44.865437+0000","last_active":"2024-04-26T21:33:31.060609+0000","last_peered":"2024-04-26T21:33:31.060609+0000","last_clean":"2024-04-26T21:33:31.060609+0000","last_became_active":"2024-04-26T21:32:44.864662+0000","last_became_peered":"2024-04-26T21:32:44.864662+0000","last_unstale":"2024-04-26T21:33:31.060609+0000","last_undegraded":"2024-04-26T21:33:31.060609+0000","last_fullsized":"2024-04-26T21:33:31.060609+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:23:37.867338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056224+0000","last_change":"2024-04-26T21:32:44.869738+0000","last_active":"2024-04-26T21:33:31.056224+0000","last_peered":"2024-04-26T21:33:31.056224+0000","last_clean":"2024-04-26T21:33:31.056224+0000","last_became_active":"2024-04-26T21:32:44.869133+0000","last_became_peered":"2024-04-26T21:32:44.869133+0000","last_unstale":"2024-04-26T21:33:31.056224+0000","last_undegraded":"2024-04-26T21:33:31.056224+0000","last_fullsized":"2024-04-26T21:33:31.056224+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:08:29.233128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056606+0000","last_change":"2024-04-26T21:32:44.856970+0000","last_active":"2024-04-26T21:33:31.056606+0000","last_peered":"2024-04-26T21:33:31.056606+0000","last_clean":"2024-04-26T21:33:31.056606+0000","last_became_active":"2024-04-26T21:32:44.856579+0000","last_became_peered":"2024-04-26T21:32:44.856579+0000","last_unstale":"2024-04-26T21:33:31.056606+0000","last_undegraded":"2024-04-26T21:33:31.056606+0000","last_fullsized":"2024-04-26T21:33:31.056606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:24:17.281094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.054658+0000","last_change":"2024-04-26T21:32:44.859688+0000","last_active":"2024-04-26T21:33:31.054658+0000","last_peered":"2024-04-26T21:33:31.054658+0000","last_clean":"2024-04-26T21:33:31.054658+0000","last_became_active":"2024-04-26T21:32:44.859336+0000","last_became_peered":"2024-04-26T21:32:44.859336+0000","last_unstale":"2024-04-26T21:33:31.054658+0000","last_undegraded":"2024-04-26T21:33:31.054658+0000","last_fullsized":"2024-04-26T21:33:31.054658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:55:46.956334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.054539+0000","last_change":"2024-04-26T21:32:44.860647+0000","last_active":"2024-04-26T21:33:31.054539+0000","last_peered":"2024-04-26T21:33:31.054539+0000","last_clean":"2024-04-26T21:33:31.054539+0000","last_became_active":"2024-04-26T21:32:44.860237+0000","last_became_peered":"2024-04-26T21:32:44.860237+0000","last_unstale":"2024-04-26T21:33:31.054539+0000","last_undegraded":"2024-04-26T21:33:31.054539+0000","last_fullsized":"2024-04-26T21:33:31.054539+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:49:10.042990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.057781+0000","last_change":"2024-04-26T21:32:44.865289+0000","last_active":"2024-04-26T21:33:31.057781+0000","last_peered":"2024-04-26T21:33:31.057781+0000","last_clean":"2024-04-26T21:33:31.057781+0000","last_became_active":"2024-04-26T21:32:44.864951+0000","last_became_peered":"2024-04-26T21:32:44.864951+0000","last_unstale":"2024-04-26T21:33:31.057781+0000","last_undegraded":"2024-04-26T21:33:31.057781+0000","last_fullsized":"2024-04-26T21:33:31.057781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:31:51.998634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.059113+0000","last_change":"2024-04-26T21:32:44.857616+0000","last_active":"2024-04-26T21:33:31.059113+0000","last_peered":"2024-04-26T21:33:31.059113+0000","last_clean":"2024-04-26T21:33:31.059113+0000","last_became_active":"2024-04-26T21:32:44.857259+0000","last_became_peered":"2024-04-26T21:32:44.857259+0000","last_unstale":"2024-04-26T21:33:31.059113+0000","last_undegraded":"2024-04-26T21:33:31.059113+0000","last_fullsized":"2024-04-26T21:33:31.059113+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:59:37.412657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.058307+0000","last_change":"2024-04-26T21:32:44.857177+0000","last_active":"2024-04-26T21:33:31.058307+0000","last_peered":"2024-04-26T21:33:31.058307+0000","last_clean":"2024-04-26T21:33:31.058307+0000","last_became_active":"2024-04-26T21:32:44.856810+0000","last_became_peered":"2024-04-26T21:32:44.856810+0000","last_unstale":"2024-04-26T21:33:31.058307+0000","last_undegraded":"2024-04-26T21:33:31.058307+0000","last_fullsized":"2024-04-26T21:33:31.058307+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:31:40.712896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060561+0000","last_change":"2024-04-26T21:32:44.870498+0000","last_active":"2024-04-26T21:33:31.060561+0000","last_peered":"2024-04-26T21:33:31.060561+0000","last_clean":"2024-04-26T21:33:31.060561+0000","last_became_active":"2024-04-26T21:32:44.870170+0000","last_became_peered":"2024-04-26T21:32:44.870170+0000","last_unstale":"2024-04-26T21:33:31.060561+0000","last_undegraded":"2024-04-26T21:33:31.060561+0000","last_fullsized":"2024-04-26T21:33:31.060561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:27.428027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.058080+0000","last_change":"2024-04-26T21:32:43.845192+0000","last_active":"2024-04-26T21:33:31.058080+0000","last_peered":"2024-04-26T21:33:31.058080+0000","last_clean":"2024-04-26T21:33:31.058080+0000","last_became_active":"2024-04-26T21:32:43.844862+0000","last_became_peered":"2024-04-26T21:32:43.844862+0000","last_unstale":"2024-04-26T21:33:31.058080+0000","last_undegraded":"2024-04-26T21:33:31.058080+0000","last_fullsized":"2024-04-26T21:33:31.058080+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:07:00.060047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.061614+0000","last_change":"2024-04-26T21:32:44.867358+0000","last_active":"2024-04-26T21:33:31.061614+0000","last_peered":"2024-04-26T21:33:31.061614+0000","last_clean":"2024-04-26T21:33:31.061614+0000","last_became_active":"2024-04-26T21:32:44.864833+0000","last_became_peered":"2024-04-26T21:32:44.864833+0000","last_unstale":"2024-04-26T21:33:31.061614+0000","last_undegraded":"2024-04-26T21:33:31.061614+0000","last_fullsized":"2024-04-26T21:33:31.061614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:45:50.285471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.061638+0000","last_change":"2024-04-26T21:32:43.849724+0000","last_active":"2024-04-26T21:33:31.061638+0000","last_peered":"2024-04-26T21:33:31.061638+0000","last_clean":"2024-04-26T21:33:31.061638+0000","last_became_active":"2024-04-26T21:32:43.849040+0000","last_became_peered":"2024-04-26T21:32:43.849040+0000","last_unstale":"2024-04-26T21:33:31.061638+0000","last_undegraded":"2024-04-26T21:33:31.061638+0000","last_fullsized":"2024-04-26T21:33:31.061638+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:48.994093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.055328+0000","last_change":"2024-04-26T21:32:44.863095+0000","last_active":"2024-04-26T21:33:31.055328+0000","last_peered":"2024-04-26T21:33:31.055328+0000","last_clean":"2024-04-26T21:33:31.055328+0000","last_became_active":"2024-04-26T21:32:44.862739+0000","last_became_peered":"2024-04-26T21:32:44.862739+0000","last_unstale":"2024-04-26T21:33:31.055328+0000","last_undegraded":"2024-04-26T21:33:31.055328+0000","last_fullsized":"2024-04-26T21:33:31.055328+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:21:19.927248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.057016+0000","last_change":"2024-04-26T21:32:43.854315+0000","last_active":"2024-04-26T21:33:31.057016+0000","last_peered":"2024-04-26T21:33:31.057016+0000","last_clean":"2024-04-26T21:33:31.057016+0000","last_became_active":"2024-04-26T21:32:43.849472+0000","last_became_peered":"2024-04-26T21:32:43.849472+0000","last_unstale":"2024-04-26T21:33:31.057016+0000","last_undegraded":"2024-04-26T21:33:31.057016+0000","last_fullsized":"2024-04-26T21:33:31.057016+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:42:24.213303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.058526+0000","last_change":"2024-04-26T21:32:44.860100+0000","last_active":"2024-04-26T21:33:31.058526+0000","last_peered":"2024-04-26T21:33:31.058526+0000","last_clean":"2024-04-26T21:33:31.058526+0000","last_became_active":"2024-04-26T21:32:44.858890+0000","last_became_peered":"2024-04-26T21:32:44.858890+0000","last_unstale":"2024-04-26T21:33:31.058526+0000","last_undegraded":"2024-04-26T21:33:31.058526+0000","last_fullsized":"2024-04-26T21:33:31.058526+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:24:22.749764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056725+0000","last_change":"2024-04-26T21:32:43.840719+0000","last_active":"2024-04-26T21:33:31.056725+0000","last_peered":"2024-04-26T21:33:31.056725+0000","last_clean":"2024-04-26T21:33:31.056725+0000","last_became_active":"2024-04-26T21:32:43.840205+0000","last_became_peered":"2024-04-26T21:32:43.840205+0000","last_unstale":"2024-04-26T21:33:31.056725+0000","last_undegraded":"2024-04-26T21:33:31.056725+0000","last_fullsized":"2024-04-26T21:33:31.056725+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:29:41.025573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.071599+0000","last_change":"2024-04-26T21:33:31.071267+0000","last_active":"2024-04-26T21:33:31.071599+0000","last_peered":"2024-04-26T21:33:31.071599+0000","last_clean":"2024-04-26T21:33:31.071599+0000","last_became_active":"2024-04-26T21:33:31.070994+0000","last_became_peered":"2024-04-26T21:33:31.070994+0000","last_unstale":"2024-04-26T21:33:31.071599+0000","last_undegraded":"2024-04-26T21:33:31.071599+0000","last_fullsized":"2024-04-26T21:33:31.071599+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:38:18.586148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,1],"acting":[9,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"85'6","reported_seq":103,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:34:03.006248+0000","last_change":"2024-04-26T21:32:43.850841+0000","last_active":"2024-04-26T21:34:03.006248+0000","last_peered":"2024-04-26T21:34:03.006248+0000","last_clean":"2024-04-26T21:34:03.006248+0000","last_became_active":"2024-04-26T21:32:43.850512+0000","last_became_peered":"2024-04-26T21:32:43.850512+0000","last_unstale":"2024-04-26T21:34:03.006248+0000","last_undegraded":"2024-04-26T21:34:03.006248+0000","last_fullsized":"2024-04-26T21:34:03.006248+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:13:51.814145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":78,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056232+0000","last_change":"2024-04-26T21:32:44.853438+0000","last_active":"2024-04-26T21:33:31.056232+0000","last_peered":"2024-04-26T21:33:31.056232+0000","last_clean":"2024-04-26T21:33:31.056232+0000","last_became_active":"2024-04-26T21:32:44.853126+0000","last_became_peered":"2024-04-26T21:32:44.853126+0000","last_unstale":"2024-04-26T21:33:31.056232+0000","last_undegraded":"2024-04-26T21:33:31.056232+0000","last_fullsized":"2024-04-26T21:33:31.056232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:43.054626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.406822+0000","last_change":"2024-04-26T21:32:43.839230+0000","last_active":"2024-04-26T21:33:31.406822+0000","last_peered":"2024-04-26T21:33:31.406822+0000","last_clean":"2024-04-26T21:33:31.406822+0000","last_became_active":"2024-04-26T21:32:43.838894+0000","last_became_peered":"2024-04-26T21:32:43.838894+0000","last_unstale":"2024-04-26T21:33:31.406822+0000","last_undegraded":"2024-04-26T21:33:31.406822+0000","last_fullsized":"2024-04-26T21:33:31.406822+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:54:29.718314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.407076+0000","last_change":"2024-04-26T21:32:44.859870+0000","last_active":"2024-04-26T21:33:31.407076+0000","last_peered":"2024-04-26T21:33:31.407076+0000","last_clean":"2024-04-26T21:33:31.407076+0000","last_became_active":"2024-04-26T21:32:44.859474+0000","last_became_peered":"2024-04-26T21:32:44.859474+0000","last_unstale":"2024-04-26T21:33:31.407076+0000","last_undegraded":"2024-04-26T21:33:31.407076+0000","last_fullsized":"2024-04-26T21:33:31.407076+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:13:14.854523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056138+0000","last_change":"2024-04-26T21:32:43.841090+0000","last_active":"2024-04-26T21:33:31.056138+0000","last_peered":"2024-04-26T21:33:31.056138+0000","last_clean":"2024-04-26T21:33:31.056138+0000","last_became_active":"2024-04-26T21:32:43.840723+0000","last_became_peered":"2024-04-26T21:32:43.840723+0000","last_unstale":"2024-04-26T21:33:31.056138+0000","last_undegraded":"2024-04-26T21:33:31.056138+0000","last_fullsized":"2024-04-26T21:33:31.056138+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:36:45.799738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.061390+0000","last_change":"2024-04-26T21:32:44.869439+0000","last_active":"2024-04-26T21:33:31.061390+0000","last_peered":"2024-04-26T21:33:31.061390+0000","last_clean":"2024-04-26T21:33:31.061390+0000","last_became_active":"2024-04-26T21:32:44.868696+0000","last_became_peered":"2024-04-26T21:32:44.868696+0000","last_unstale":"2024-04-26T21:33:31.061390+0000","last_undegraded":"2024-04-26T21:33:31.061390+0000","last_fullsized":"2024-04-26T21:33:31.061390+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:14:46.400937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.406739+0000","last_change":"2024-04-26T21:32:43.852431+0000","last_active":"2024-04-26T21:33:31.406739+0000","last_peered":"2024-04-26T21:33:31.406739+0000","last_clean":"2024-04-26T21:33:31.406739+0000","last_became_active":"2024-04-26T21:32:43.852143+0000","last_became_peered":"2024-04-26T21:32:43.852143+0000","last_unstale":"2024-04-26T21:33:31.406739+0000","last_undegraded":"2024-04-26T21:33:31.406739+0000","last_fullsized":"2024-04-26T21:33:31.406739+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:35.971582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.057707+0000","last_change":"2024-04-26T21:32:44.861118+0000","last_active":"2024-04-26T21:33:31.057707+0000","last_peered":"2024-04-26T21:33:31.057707+0000","last_clean":"2024-04-26T21:33:31.057707+0000","last_became_active":"2024-04-26T21:32:44.860765+0000","last_became_peered":"2024-04-26T21:32:44.860765+0000","last_unstale":"2024-04-26T21:33:31.057707+0000","last_undegraded":"2024-04-26T21:33:31.057707+0000","last_fullsized":"2024-04-26T21:33:31.057707+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:01:09.370401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":97,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:34:03.005362+0000","last_change":"2024-04-26T21:32:43.840267+0000","last_active":"2024-04-26T21:34:03.005362+0000","last_peered":"2024-04-26T21:34:03.005362+0000","last_clean":"2024-04-26T21:34:03.005362+0000","last_became_active":"2024-04-26T21:32:43.839944+0000","last_became_peered":"2024-04-26T21:32:43.839944+0000","last_unstale":"2024-04-26T21:34:03.005362+0000","last_undegraded":"2024-04-26T21:34:03.005362+0000","last_fullsized":"2024-04-26T21:34:03.005362+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:46:13.671063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":75,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.058451+0000","last_change":"2024-04-26T21:32:44.861772+0000","last_active":"2024-04-26T21:33:31.058451+0000","last_peered":"2024-04-26T21:33:31.058451+0000","last_clean":"2024-04-26T21:33:31.058451+0000","last_became_active":"2024-04-26T21:32:44.860376+0000","last_became_peered":"2024-04-26T21:32:44.860376+0000","last_unstale":"2024-04-26T21:33:31.058451+0000","last_undegraded":"2024-04-26T21:33:31.058451+0000","last_fullsized":"2024-04-26T21:33:31.058451+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:07:29.499706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060319+0000","last_change":"2024-04-26T21:32:43.849923+0000","last_active":"2024-04-26T21:33:31.060319+0000","last_peered":"2024-04-26T21:33:31.060319+0000","last_clean":"2024-04-26T21:33:31.060319+0000","last_became_active":"2024-04-26T21:32:43.848193+0000","last_became_peered":"2024-04-26T21:32:43.848193+0000","last_unstale":"2024-04-26T21:33:31.060319+0000","last_undegraded":"2024-04-26T21:33:31.060319+0000","last_fullsized":"2024-04-26T21:33:31.060319+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:06:00.247084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.055308+0000","last_change":"2024-04-26T21:32:44.866684+0000","last_active":"2024-04-26T21:33:31.055308+0000","last_peered":"2024-04-26T21:33:31.055308+0000","last_clean":"2024-04-26T21:33:31.055308+0000","last_became_active":"2024-04-26T21:32:44.866350+0000","last_became_peered":"2024-04-26T21:32:44.866350+0000","last_unstale":"2024-04-26T21:33:31.055308+0000","last_undegraded":"2024-04-26T21:33:31.055308+0000","last_fullsized":"2024-04-26T21:33:31.055308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:56:21.306500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.407202+0000","last_change":"2024-04-26T21:32:43.841617+0000","last_active":"2024-04-26T21:33:31.407202+0000","last_peered":"2024-04-26T21:33:31.407202+0000","last_clean":"2024-04-26T21:33:31.407202+0000","last_became_active":"2024-04-26T21:32:43.841302+0000","last_became_peered":"2024-04-26T21:32:43.841302+0000","last_unstale":"2024-04-26T21:33:31.407202+0000","last_undegraded":"2024-04-26T21:33:31.407202+0000","last_fullsized":"2024-04-26T21:33:31.407202+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:16:25.409255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.061757+0000","last_change":"2024-04-26T21:33:31.061757+0000","last_active":"2024-04-26T21:33:31.061757+0000","last_peered":"2024-04-26T21:33:31.061757+0000","last_clean":"2024-04-26T21:33:31.061757+0000","last_became_active":"2024-04-26T21:33:31.061300+0000","last_became_peered":"2024-04-26T21:33:31.061300+0000","last_unstale":"2024-04-26T21:33:31.061757+0000","last_undegraded":"2024-04-26T21:33:31.061757+0000","last_fullsized":"2024-04-26T21:33:31.061757+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:14:41.011436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'25","reported_seq":201,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:34:04.020448+0000","last_change":"2024-04-26T21:32:43.843122+0000","last_active":"2024-04-26T21:34:04.020448+0000","last_peered":"2024-04-26T21:34:04.020448+0000","last_clean":"2024-04-26T21:34:04.020448+0000","last_became_active":"2024-04-26T21:32:43.842751+0000","last_became_peered":"2024-04-26T21:32:43.842751+0000","last_unstale":"2024-04-26T21:34:04.020448+0000","last_undegraded":"2024-04-26T21:34:04.020448+0000","last_fullsized":"2024-04-26T21:34:04.020448+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:16:58.533030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49477,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":157,"num_read_kb":126,"num_write":26,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056599+0000","last_change":"2024-04-26T21:32:44.872829+0000","last_active":"2024-04-26T21:33:31.056599+0000","last_peered":"2024-04-26T21:33:31.056599+0000","last_clean":"2024-04-26T21:33:31.056599+0000","last_became_active":"2024-04-26T21:32:44.872466+0000","last_became_peered":"2024-04-26T21:32:44.872466+0000","last_unstale":"2024-04-26T21:33:31.056599+0000","last_undegraded":"2024-04-26T21:33:31.056599+0000","last_fullsized":"2024-04-26T21:33:31.056599+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:07:27.698789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060409+0000","last_change":"2024-04-26T21:32:43.840277+0000","last_active":"2024-04-26T21:33:31.060409+0000","last_peered":"2024-04-26T21:33:31.060409+0000","last_clean":"2024-04-26T21:33:31.060409+0000","last_became_active":"2024-04-26T21:32:43.839962+0000","last_became_peered":"2024-04-26T21:32:43.839962+0000","last_unstale":"2024-04-26T21:33:31.060409+0000","last_undegraded":"2024-04-26T21:33:31.060409+0000","last_fullsized":"2024-04-26T21:33:31.060409+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:25:54.475216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056357+0000","last_change":"2024-04-26T21:32:44.866465+0000","last_active":"2024-04-26T21:33:31.056357+0000","last_peered":"2024-04-26T21:33:31.056357+0000","last_clean":"2024-04-26T21:33:31.056357+0000","last_became_active":"2024-04-26T21:32:44.866067+0000","last_became_peered":"2024-04-26T21:32:44.866067+0000","last_unstale":"2024-04-26T21:33:31.056357+0000","last_undegraded":"2024-04-26T21:33:31.056357+0000","last_fullsized":"2024-04-26T21:33:31.056357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:47:03.884436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.406841+0000","last_change":"2024-04-26T21:32:43.851535+0000","last_active":"2024-04-26T21:33:31.406841+0000","last_peered":"2024-04-26T21:33:31.406841+0000","last_clean":"2024-04-26T21:33:31.406841+0000","last_became_active":"2024-04-26T21:32:43.851237+0000","last_became_peered":"2024-04-26T21:32:43.851237+0000","last_unstale":"2024-04-26T21:33:31.406841+0000","last_undegraded":"2024-04-26T21:33:31.406841+0000","last_fullsized":"2024-04-26T21:33:31.406841+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:08:40.624027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060685+0000","last_change":"2024-04-26T21:32:44.874686+0000","last_active":"2024-04-26T21:33:31.060685+0000","last_peered":"2024-04-26T21:33:31.060685+0000","last_clean":"2024-04-26T21:33:31.060685+0000","last_became_active":"2024-04-26T21:32:44.874377+0000","last_became_peered":"2024-04-26T21:32:44.874377+0000","last_unstale":"2024-04-26T21:33:31.060685+0000","last_undegraded":"2024-04-26T21:33:31.060685+0000","last_fullsized":"2024-04-26T21:33:31.060685+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:45:33.627663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'4","reported_seq":101,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:34:03.005206+0000","last_change":"2024-04-26T21:32:43.846522+0000","last_active":"2024-04-26T21:34:03.005206+0000","last_peered":"2024-04-26T21:34:03.005206+0000","last_clean":"2024-04-26T21:34:03.005206+0000","last_became_active":"2024-04-26T21:32:43.846138+0000","last_became_peered":"2024-04-26T21:32:43.846138+0000","last_unstale":"2024-04-26T21:34:03.005206+0000","last_undegraded":"2024-04-26T21:34:03.005206+0000","last_fullsized":"2024-04-26T21:34:03.005206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:28:40.257244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060034+0000","last_change":"2024-04-26T21:32:44.872228+0000","last_active":"2024-04-26T21:33:31.060034+0000","last_peered":"2024-04-26T21:33:31.060034+0000","last_clean":"2024-04-26T21:33:31.060034+0000","last_became_active":"2024-04-26T21:32:44.871818+0000","last_became_peered":"2024-04-26T21:32:44.871818+0000","last_unstale":"2024-04-26T21:33:31.060034+0000","last_undegraded":"2024-04-26T21:33:31.060034+0000","last_fullsized":"2024-04-26T21:33:31.060034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:52:32.520802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":59,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.073977+0000","last_change":"2024-04-26T21:33:31.073977+0000","last_active":"2024-04-26T21:33:31.073977+0000","last_peered":"2024-04-26T21:33:31.073977+0000","last_clean":"2024-04-26T21:33:31.073977+0000","last_became_active":"2024-04-26T21:33:31.060006+0000","last_became_peered":"2024-04-26T21:33:31.060006+0000","last_unstale":"2024-04-26T21:33:31.073977+0000","last_undegraded":"2024-04-26T21:33:31.073977+0000","last_fullsized":"2024-04-26T21:33:31.073977+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:13:23.795696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.406782+0000","last_change":"2024-04-26T21:32:44.859853+0000","last_active":"2024-04-26T21:33:31.406782+0000","last_peered":"2024-04-26T21:33:31.406782+0000","last_clean":"2024-04-26T21:33:31.406782+0000","last_became_active":"2024-04-26T21:32:44.859521+0000","last_became_peered":"2024-04-26T21:32:44.859521+0000","last_unstale":"2024-04-26T21:33:31.406782+0000","last_undegraded":"2024-04-26T21:33:31.406782+0000","last_fullsized":"2024-04-26T21:33:31.406782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:48:54.955913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056949+0000","last_change":"2024-04-26T21:32:43.840795+0000","last_active":"2024-04-26T21:33:31.056949+0000","last_peered":"2024-04-26T21:33:31.056949+0000","last_clean":"2024-04-26T21:33:31.056949+0000","last_became_active":"2024-04-26T21:32:43.840443+0000","last_became_peered":"2024-04-26T21:32:43.840443+0000","last_unstale":"2024-04-26T21:33:31.056949+0000","last_undegraded":"2024-04-26T21:33:31.056949+0000","last_fullsized":"2024-04-26T21:33:31.056949+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:09:48.944085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.058755+0000","last_change":"2024-04-26T21:32:44.862344+0000","last_active":"2024-04-26T21:33:31.058755+0000","last_peered":"2024-04-26T21:33:31.058755+0000","last_clean":"2024-04-26T21:33:31.058755+0000","last_became_active":"2024-04-26T21:32:44.862029+0000","last_became_peered":"2024-04-26T21:32:44.862029+0000","last_unstale":"2024-04-26T21:33:31.058755+0000","last_undegraded":"2024-04-26T21:33:31.058755+0000","last_fullsized":"2024-04-26T21:33:31.058755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:45:48.833753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.058095+0000","last_change":"2024-04-26T21:32:43.843853+0000","last_active":"2024-04-26T21:33:31.058095+0000","last_peered":"2024-04-26T21:33:31.058095+0000","last_clean":"2024-04-26T21:33:31.058095+0000","last_became_active":"2024-04-26T21:32:43.843525+0000","last_became_peered":"2024-04-26T21:32:43.843525+0000","last_unstale":"2024-04-26T21:33:31.058095+0000","last_undegraded":"2024-04-26T21:33:31.058095+0000","last_fullsized":"2024-04-26T21:33:31.058095+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:11:43.222849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.407172+0000","last_change":"2024-04-26T21:32:44.869123+0000","last_active":"2024-04-26T21:33:31.407172+0000","last_peered":"2024-04-26T21:33:31.407172+0000","last_clean":"2024-04-26T21:33:31.407172+0000","last_became_active":"2024-04-26T21:32:44.868573+0000","last_became_peered":"2024-04-26T21:32:44.868573+0000","last_unstale":"2024-04-26T21:33:31.407172+0000","last_undegraded":"2024-04-26T21:33:31.407172+0000","last_fullsized":"2024-04-26T21:33:31.407172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:48:54.447157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060563+0000","last_change":"2024-04-26T21:32:43.850596+0000","last_active":"2024-04-26T21:33:31.060563+0000","last_peered":"2024-04-26T21:33:31.060563+0000","last_clean":"2024-04-26T21:33:31.060563+0000","last_became_active":"2024-04-26T21:32:43.850218+0000","last_became_peered":"2024-04-26T21:32:43.850218+0000","last_unstale":"2024-04-26T21:33:31.060563+0000","last_undegraded":"2024-04-26T21:33:31.060563+0000","last_fullsized":"2024-04-26T21:33:31.060563+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:38:34.068783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.406954+0000","last_change":"2024-04-26T21:32:44.863450+0000","last_active":"2024-04-26T21:33:31.406954+0000","last_peered":"2024-04-26T21:33:31.406954+0000","last_clean":"2024-04-26T21:33:31.406954+0000","last_became_active":"2024-04-26T21:32:44.863089+0000","last_became_peered":"2024-04-26T21:32:44.863089+0000","last_unstale":"2024-04-26T21:33:31.406954+0000","last_undegraded":"2024-04-26T21:33:31.406954+0000","last_fullsized":"2024-04-26T21:33:31.406954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:04:51.821073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.406892+0000","last_change":"2024-04-26T21:32:44.855575+0000","last_active":"2024-04-26T21:33:31.406892+0000","last_peered":"2024-04-26T21:33:31.406892+0000","last_clean":"2024-04-26T21:33:31.406892+0000","last_became_active":"2024-04-26T21:32:44.855258+0000","last_became_peered":"2024-04-26T21:32:44.855258+0000","last_unstale":"2024-04-26T21:33:31.406892+0000","last_undegraded":"2024-04-26T21:33:31.406892+0000","last_fullsized":"2024-04-26T21:33:31.406892+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:41:07.559095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":98,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:34:03.004293+0000","last_change":"2024-04-26T21:32:43.850713+0000","last_active":"2024-04-26T21:34:03.004293+0000","last_peered":"2024-04-26T21:34:03.004293+0000","last_clean":"2024-04-26T21:34:03.004293+0000","last_became_active":"2024-04-26T21:32:43.850406+0000","last_became_peered":"2024-04-26T21:32:43.850406+0000","last_unstale":"2024-04-26T21:34:03.004293+0000","last_undegraded":"2024-04-26T21:34:03.004293+0000","last_fullsized":"2024-04-26T21:34:03.004293+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:11:02.682963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":75,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.061377+0000","last_change":"2024-04-26T21:32:44.868317+0000","last_active":"2024-04-26T21:33:31.061377+0000","last_peered":"2024-04-26T21:33:31.061377+0000","last_clean":"2024-04-26T21:33:31.061377+0000","last_became_active":"2024-04-26T21:32:44.864571+0000","last_became_peered":"2024-04-26T21:32:44.864571+0000","last_unstale":"2024-04-26T21:33:31.061377+0000","last_undegraded":"2024-04-26T21:33:31.061377+0000","last_fullsized":"2024-04-26T21:33:31.061377+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:24:50.476949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056133+0000","last_change":"2024-04-26T21:32:43.833773+0000","last_active":"2024-04-26T21:33:31.056133+0000","last_peered":"2024-04-26T21:33:31.056133+0000","last_clean":"2024-04-26T21:33:31.056133+0000","last_became_active":"2024-04-26T21:32:43.833338+0000","last_became_peered":"2024-04-26T21:32:43.833338+0000","last_unstale":"2024-04-26T21:33:31.056133+0000","last_undegraded":"2024-04-26T21:33:31.056133+0000","last_fullsized":"2024-04-26T21:33:31.056133+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:17:15.382494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'137","reported_seq":340,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060158+0000","last_change":"2024-04-26T21:31:27.930370+0000","last_active":"2024-04-26T21:33:31.060158+0000","last_peered":"2024-04-26T21:33:31.060158+0000","last_clean":"2024-04-26T21:33:31.060158+0000","last_became_active":"2024-04-26T21:31:27.929859+0000","last_became_peered":"2024-04-26T21:31:27.929859+0000","last_unstale":"2024-04-26T21:33:31.060158+0000","last_undegraded":"2024-04-26T21:33:31.060158+0000","last_fullsized":"2024-04-26T21:33:31.060158+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:58.528699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:58.528699+0000","last_clean_scrub_stamp":"2024-04-26T21:27:58.528699+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:00:55.701865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.055991+0000","last_change":"2024-04-26T21:32:44.858891+0000","last_active":"2024-04-26T21:33:31.055991+0000","last_peered":"2024-04-26T21:33:31.055991+0000","last_clean":"2024-04-26T21:33:31.055991+0000","last_became_active":"2024-04-26T21:32:44.858406+0000","last_became_peered":"2024-04-26T21:32:44.858406+0000","last_unstale":"2024-04-26T21:33:31.055991+0000","last_undegraded":"2024-04-26T21:33:31.055991+0000","last_fullsized":"2024-04-26T21:33:31.055991+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:40:13.084352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060332+0000","last_change":"2024-04-26T21:32:43.849442+0000","last_active":"2024-04-26T21:33:31.060332+0000","last_peered":"2024-04-26T21:33:31.060332+0000","last_clean":"2024-04-26T21:33:31.060332+0000","last_became_active":"2024-04-26T21:32:43.847951+0000","last_became_peered":"2024-04-26T21:32:43.847951+0000","last_unstale":"2024-04-26T21:33:31.060332+0000","last_undegraded":"2024-04-26T21:33:31.060332+0000","last_fullsized":"2024-04-26T21:33:31.060332+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:55:00.025931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056679+0000","last_change":"2024-04-26T21:32:43.846736+0000","last_active":"2024-04-26T21:33:31.056679+0000","last_peered":"2024-04-26T21:33:31.056679+0000","last_clean":"2024-04-26T21:33:31.056679+0000","last_became_active":"2024-04-26T21:32:43.846451+0000","last_became_peered":"2024-04-26T21:32:43.846451+0000","last_unstale":"2024-04-26T21:33:31.056679+0000","last_undegraded":"2024-04-26T21:33:31.056679+0000","last_fullsized":"2024-04-26T21:33:31.056679+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:38:46.301597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.057016+0000","last_change":"2024-04-26T21:32:44.856817+0000","last_active":"2024-04-26T21:33:31.057016+0000","last_peered":"2024-04-26T21:33:31.057016+0000","last_clean":"2024-04-26T21:33:31.057016+0000","last_became_active":"2024-04-26T21:32:44.856489+0000","last_became_peered":"2024-04-26T21:32:44.856489+0000","last_unstale":"2024-04-26T21:33:31.057016+0000","last_undegraded":"2024-04-26T21:33:31.057016+0000","last_fullsized":"2024-04-26T21:33:31.057016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:13:33.851351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"87'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.058457+0000","last_change":"2024-04-26T21:32:43.838974+0000","last_active":"2024-04-26T21:33:31.058457+0000","last_peered":"2024-04-26T21:33:31.058457+0000","last_clean":"2024-04-26T21:33:31.058457+0000","last_became_active":"2024-04-26T21:32:43.838661+0000","last_became_peered":"2024-04-26T21:32:43.838661+0000","last_unstale":"2024-04-26T21:33:31.058457+0000","last_undegraded":"2024-04-26T21:33:31.058457+0000","last_fullsized":"2024-04-26T21:33:31.058457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:44:29.536559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.061606+0000","last_change":"2024-04-26T21:32:44.872716+0000","last_active":"2024-04-26T21:33:31.061606+0000","last_peered":"2024-04-26T21:33:31.061606+0000","last_clean":"2024-04-26T21:33:31.061606+0000","last_became_active":"2024-04-26T21:32:44.872348+0000","last_became_peered":"2024-04-26T21:32:44.872348+0000","last_unstale":"2024-04-26T21:33:31.061606+0000","last_undegraded":"2024-04-26T21:33:31.061606+0000","last_fullsized":"2024-04-26T21:33:31.061606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:17:56.890488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056380+0000","last_change":"2024-04-26T21:32:43.853138+0000","last_active":"2024-04-26T21:33:31.056380+0000","last_peered":"2024-04-26T21:33:31.056380+0000","last_clean":"2024-04-26T21:33:31.056380+0000","last_became_active":"2024-04-26T21:32:43.852803+0000","last_became_peered":"2024-04-26T21:32:43.852803+0000","last_unstale":"2024-04-26T21:33:31.056380+0000","last_undegraded":"2024-04-26T21:33:31.056380+0000","last_fullsized":"2024-04-26T21:33:31.056380+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:59:04.790442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.055193+0000","last_change":"2024-04-26T21:32:44.867559+0000","last_active":"2024-04-26T21:33:31.055193+0000","last_peered":"2024-04-26T21:33:31.055193+0000","last_clean":"2024-04-26T21:33:31.055193+0000","last_became_active":"2024-04-26T21:32:44.867229+0000","last_became_peered":"2024-04-26T21:32:44.867229+0000","last_unstale":"2024-04-26T21:33:31.055193+0000","last_undegraded":"2024-04-26T21:33:31.055193+0000","last_fullsized":"2024-04-26T21:33:31.055193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:09:09.052753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.057864+0000","last_change":"2024-04-26T21:32:43.844761+0000","last_active":"2024-04-26T21:33:31.057864+0000","last_peered":"2024-04-26T21:33:31.057864+0000","last_clean":"2024-04-26T21:33:31.057864+0000","last_became_active":"2024-04-26T21:32:43.844257+0000","last_became_peered":"2024-04-26T21:32:43.844257+0000","last_unstale":"2024-04-26T21:33:31.057864+0000","last_undegraded":"2024-04-26T21:33:31.057864+0000","last_fullsized":"2024-04-26T21:33:31.057864+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:06:55.560467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056125+0000","last_change":"2024-04-26T21:32:44.856662+0000","last_active":"2024-04-26T21:33:31.056125+0000","last_peered":"2024-04-26T21:33:31.056125+0000","last_clean":"2024-04-26T21:33:31.056125+0000","last_became_active":"2024-04-26T21:32:44.856297+0000","last_became_peered":"2024-04-26T21:32:44.856297+0000","last_unstale":"2024-04-26T21:33:31.056125+0000","last_undegraded":"2024-04-26T21:33:31.056125+0000","last_fullsized":"2024-04-26T21:33:31.056125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:48:55.314959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060803+0000","last_change":"2024-04-26T21:32:43.848710+0000","last_active":"2024-04-26T21:33:31.060803+0000","last_peered":"2024-04-26T21:33:31.060803+0000","last_clean":"2024-04-26T21:33:31.060803+0000","last_became_active":"2024-04-26T21:32:43.848379+0000","last_became_peered":"2024-04-26T21:32:43.848379+0000","last_unstale":"2024-04-26T21:33:31.060803+0000","last_undegraded":"2024-04-26T21:33:31.060803+0000","last_fullsized":"2024-04-26T21:33:31.060803+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:26:44.233330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.058122+0000","last_change":"2024-04-26T21:32:44.858327+0000","last_active":"2024-04-26T21:33:31.058122+0000","last_peered":"2024-04-26T21:33:31.058122+0000","last_clean":"2024-04-26T21:33:31.058122+0000","last_became_active":"2024-04-26T21:32:44.858023+0000","last_became_peered":"2024-04-26T21:32:44.858023+0000","last_unstale":"2024-04-26T21:33:31.058122+0000","last_undegraded":"2024-04-26T21:33:31.058122+0000","last_fullsized":"2024-04-26T21:33:31.058122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:37:30.588063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.068501+0000","last_change":"2024-04-26T21:33:31.068501+0000","last_active":"2024-04-26T21:33:31.068501+0000","last_peered":"2024-04-26T21:33:31.068501+0000","last_clean":"2024-04-26T21:33:31.068501+0000","last_became_active":"2024-04-26T21:33:31.063005+0000","last_became_peered":"2024-04-26T21:33:31.063005+0000","last_unstale":"2024-04-26T21:33:31.068501+0000","last_undegraded":"2024-04-26T21:33:31.068501+0000","last_fullsized":"2024-04-26T21:33:31.068501+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:50:04.843538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.061020+0000","last_change":"2024-04-26T21:32:44.867088+0000","last_active":"2024-04-26T21:33:31.061020+0000","last_peered":"2024-04-26T21:33:31.061020+0000","last_clean":"2024-04-26T21:33:31.061020+0000","last_became_active":"2024-04-26T21:32:44.863623+0000","last_became_peered":"2024-04-26T21:32:44.863623+0000","last_unstale":"2024-04-26T21:33:31.061020+0000","last_undegraded":"2024-04-26T21:33:31.061020+0000","last_fullsized":"2024-04-26T21:33:31.061020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:42:21.107362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.066518+0000","last_change":"2024-04-26T21:33:31.066518+0000","last_active":"2024-04-26T21:33:31.066518+0000","last_peered":"2024-04-26T21:33:31.066518+0000","last_clean":"2024-04-26T21:33:31.066518+0000","last_became_active":"2024-04-26T21:33:31.061769+0000","last_became_peered":"2024-04-26T21:33:31.061769+0000","last_unstale":"2024-04-26T21:33:31.066518+0000","last_undegraded":"2024-04-26T21:33:31.066518+0000","last_fullsized":"2024-04-26T21:33:31.066518+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:56:30.724712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060839+0000","last_change":"2024-04-26T21:32:44.863365+0000","last_active":"2024-04-26T21:33:31.060839+0000","last_peered":"2024-04-26T21:33:31.060839+0000","last_clean":"2024-04-26T21:33:31.060839+0000","last_became_active":"2024-04-26T21:32:44.863012+0000","last_became_peered":"2024-04-26T21:32:44.863012+0000","last_unstale":"2024-04-26T21:33:31.060839+0000","last_undegraded":"2024-04-26T21:33:31.060839+0000","last_fullsized":"2024-04-26T21:33:31.060839+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:09:50.023773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060732+0000","last_change":"2024-04-26T21:32:43.853507+0000","last_active":"2024-04-26T21:33:31.060732+0000","last_peered":"2024-04-26T21:33:31.060732+0000","last_clean":"2024-04-26T21:33:31.060732+0000","last_became_active":"2024-04-26T21:32:43.852680+0000","last_became_peered":"2024-04-26T21:32:43.852680+0000","last_unstale":"2024-04-26T21:33:31.060732+0000","last_undegraded":"2024-04-26T21:33:31.060732+0000","last_fullsized":"2024-04-26T21:33:31.060732+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:46:51.836736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060622+0000","last_change":"2024-04-26T21:32:44.865158+0000","last_active":"2024-04-26T21:33:31.060622+0000","last_peered":"2024-04-26T21:33:31.060622+0000","last_clean":"2024-04-26T21:33:31.060622+0000","last_became_active":"2024-04-26T21:32:44.864182+0000","last_became_peered":"2024-04-26T21:32:44.864182+0000","last_unstale":"2024-04-26T21:33:31.060622+0000","last_undegraded":"2024-04-26T21:33:31.060622+0000","last_fullsized":"2024-04-26T21:33:31.060622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:44:55.918623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.407171+0000","last_change":"2024-04-26T21:32:43.842913+0000","last_active":"2024-04-26T21:33:31.407171+0000","last_peered":"2024-04-26T21:33:31.407171+0000","last_clean":"2024-04-26T21:33:31.407171+0000","last_became_active":"2024-04-26T21:32:43.842599+0000","last_became_peered":"2024-04-26T21:32:43.842599+0000","last_unstale":"2024-04-26T21:33:31.407171+0000","last_undegraded":"2024-04-26T21:33:31.407171+0000","last_fullsized":"2024-04-26T21:33:31.407171+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:46.583398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060236+0000","last_change":"2024-04-26T21:32:44.864239+0000","last_active":"2024-04-26T21:33:31.060236+0000","last_peered":"2024-04-26T21:33:31.060236+0000","last_clean":"2024-04-26T21:33:31.060236+0000","last_became_active":"2024-04-26T21:32:44.863862+0000","last_became_peered":"2024-04-26T21:32:44.863862+0000","last_unstale":"2024-04-26T21:33:31.060236+0000","last_undegraded":"2024-04-26T21:33:31.060236+0000","last_fullsized":"2024-04-26T21:33:31.060236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:32:16.958044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056742+0000","last_change":"2024-04-26T21:32:43.849768+0000","last_active":"2024-04-26T21:33:31.056742+0000","last_peered":"2024-04-26T21:33:31.056742+0000","last_clean":"2024-04-26T21:33:31.056742+0000","last_became_active":"2024-04-26T21:32:43.849424+0000","last_became_peered":"2024-04-26T21:32:43.849424+0000","last_unstale":"2024-04-26T21:33:31.056742+0000","last_undegraded":"2024-04-26T21:33:31.056742+0000","last_fullsized":"2024-04-26T21:33:31.056742+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:42.820059+0000","last_clean_scrub_stamp":"2024-04-26T21:32:42.820059+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:46:23.231819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060448+0000","last_change":"2024-04-26T21:32:44.862597+0000","last_active":"2024-04-26T21:33:31.060448+0000","last_peered":"2024-04-26T21:33:31.060448+0000","last_clean":"2024-04-26T21:33:31.060448+0000","last_became_active":"2024-04-26T21:32:44.862259+0000","last_became_peered":"2024-04-26T21:32:44.862259+0000","last_unstale":"2024-04-26T21:33:31.060448+0000","last_undegraded":"2024-04-26T21:33:31.060448+0000","last_fullsized":"2024-04-26T21:33:31.060448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:49:19.759329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.058228+0000","last_change":"2024-04-26T21:32:44.855196+0000","last_active":"2024-04-26T21:33:31.058228+0000","last_peered":"2024-04-26T21:33:31.058228+0000","last_clean":"2024-04-26T21:33:31.058228+0000","last_became_active":"2024-04-26T21:32:44.854774+0000","last_became_peered":"2024-04-26T21:32:44.854774+0000","last_unstale":"2024-04-26T21:33:31.058228+0000","last_undegraded":"2024-04-26T21:33:31.058228+0000","last_fullsized":"2024-04-26T21:33:31.058228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:46:18.370983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060704+0000","last_change":"2024-04-26T21:32:44.868115+0000","last_active":"2024-04-26T21:33:31.060704+0000","last_peered":"2024-04-26T21:33:31.060704+0000","last_clean":"2024-04-26T21:33:31.060704+0000","last_became_active":"2024-04-26T21:32:44.867767+0000","last_became_peered":"2024-04-26T21:32:44.867767+0000","last_unstale":"2024-04-26T21:33:31.060704+0000","last_undegraded":"2024-04-26T21:33:31.060704+0000","last_fullsized":"2024-04-26T21:33:31.060704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:30:22.130786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060399+0000","last_change":"2024-04-26T21:32:44.864633+0000","last_active":"2024-04-26T21:33:31.060399+0000","last_peered":"2024-04-26T21:33:31.060399+0000","last_clean":"2024-04-26T21:33:31.060399+0000","last_became_active":"2024-04-26T21:32:44.864245+0000","last_became_peered":"2024-04-26T21:32:44.864245+0000","last_unstale":"2024-04-26T21:33:31.060399+0000","last_undegraded":"2024-04-26T21:33:31.060399+0000","last_fullsized":"2024-04-26T21:33:31.060399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:10:20.315396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.054775+0000","last_change":"2024-04-26T21:32:44.858122+0000","last_active":"2024-04-26T21:33:31.054775+0000","last_peered":"2024-04-26T21:33:31.054775+0000","last_clean":"2024-04-26T21:33:31.054775+0000","last_became_active":"2024-04-26T21:32:44.857513+0000","last_became_peered":"2024-04-26T21:32:44.857513+0000","last_unstale":"2024-04-26T21:33:31.054775+0000","last_undegraded":"2024-04-26T21:33:31.054775+0000","last_fullsized":"2024-04-26T21:33:31.054775+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:50:40.725215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.059869+0000","last_change":"2024-04-26T21:33:31.059869+0000","last_active":"2024-04-26T21:33:31.059869+0000","last_peered":"2024-04-26T21:33:31.059869+0000","last_clean":"2024-04-26T21:33:31.059869+0000","last_became_active":"2024-04-26T21:33:31.059581+0000","last_became_peered":"2024-04-26T21:33:31.059581+0000","last_unstale":"2024-04-26T21:33:31.059869+0000","last_undegraded":"2024-04-26T21:33:31.059869+0000","last_fullsized":"2024-04-26T21:33:31.059869+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:05:21.289959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.059005+0000","last_change":"2024-04-26T21:32:44.852073+0000","last_active":"2024-04-26T21:33:31.059005+0000","last_peered":"2024-04-26T21:33:31.059005+0000","last_clean":"2024-04-26T21:33:31.059005+0000","last_became_active":"2024-04-26T21:32:44.851717+0000","last_became_peered":"2024-04-26T21:32:44.851717+0000","last_unstale":"2024-04-26T21:33:31.059005+0000","last_undegraded":"2024-04-26T21:33:31.059005+0000","last_fullsized":"2024-04-26T21:33:31.059005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:27:37.599933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060852+0000","last_change":"2024-04-26T21:32:44.859503+0000","last_active":"2024-04-26T21:33:31.060852+0000","last_peered":"2024-04-26T21:33:31.060852+0000","last_clean":"2024-04-26T21:33:31.060852+0000","last_became_active":"2024-04-26T21:32:44.859009+0000","last_became_peered":"2024-04-26T21:32:44.859009+0000","last_unstale":"2024-04-26T21:33:31.060852+0000","last_undegraded":"2024-04-26T21:33:31.060852+0000","last_fullsized":"2024-04-26T21:33:31.060852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:00:26.307567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.062113+0000","last_change":"2024-04-26T21:33:31.061589+0000","last_active":"2024-04-26T21:33:31.062113+0000","last_peered":"2024-04-26T21:33:31.062113+0000","last_clean":"2024-04-26T21:33:31.062113+0000","last_became_active":"2024-04-26T21:33:31.061277+0000","last_became_peered":"2024-04-26T21:33:31.061277+0000","last_unstale":"2024-04-26T21:33:31.062113+0000","last_undegraded":"2024-04-26T21:33:31.062113+0000","last_fullsized":"2024-04-26T21:33:31.062113+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:36:40.775558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,2],"acting":[9,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.406820+0000","last_change":"2024-04-26T21:32:44.857666+0000","last_active":"2024-04-26T21:33:31.406820+0000","last_peered":"2024-04-26T21:33:31.406820+0000","last_clean":"2024-04-26T21:33:31.406820+0000","last_became_active":"2024-04-26T21:32:44.857342+0000","last_became_peered":"2024-04-26T21:32:44.857342+0000","last_unstale":"2024-04-26T21:33:31.406820+0000","last_undegraded":"2024-04-26T21:33:31.406820+0000","last_fullsized":"2024-04-26T21:33:31.406820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:05:15.139919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.060911+0000","last_change":"2024-04-26T21:32:44.868571+0000","last_active":"2024-04-26T21:33:31.060911+0000","last_peered":"2024-04-26T21:33:31.060911+0000","last_clean":"2024-04-26T21:33:31.060911+0000","last_became_active":"2024-04-26T21:32:44.868191+0000","last_became_peered":"2024-04-26T21:32:44.868191+0000","last_unstale":"2024-04-26T21:33:31.060911+0000","last_undegraded":"2024-04-26T21:33:31.060911+0000","last_fullsized":"2024-04-26T21:33:31.060911+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:30:35.329277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.058384+0000","last_change":"2024-04-26T21:32:44.867823+0000","last_active":"2024-04-26T21:33:31.058384+0000","last_peered":"2024-04-26T21:33:31.058384+0000","last_clean":"2024-04-26T21:33:31.058384+0000","last_became_active":"2024-04-26T21:32:44.867496+0000","last_became_peered":"2024-04-26T21:32:44.867496+0000","last_unstale":"2024-04-26T21:33:31.058384+0000","last_undegraded":"2024-04-26T21:33:31.058384+0000","last_fullsized":"2024-04-26T21:33:31.058384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:41:47.484332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.056623+0000","last_change":"2024-04-26T21:32:44.864720+0000","last_active":"2024-04-26T21:33:31.056623+0000","last_peered":"2024-04-26T21:33:31.056623+0000","last_clean":"2024-04-26T21:33:31.056623+0000","last_became_active":"2024-04-26T21:32:44.864395+0000","last_became_peered":"2024-04-26T21:32:44.864395+0000","last_unstale":"2024-04-26T21:33:31.056623+0000","last_undegraded":"2024-04-26T21:33:31.056623+0000","last_fullsized":"2024-04-26T21:33:31.056623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:56:25.416249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-04-26T21:33:31.058608+0000","last_change":"2024-04-26T21:32:44.862415+0000","last_active":"2024-04-26T21:33:31.058608+0000","last_peered":"2024-04-26T21:33:31.058608+0000","last_clean":"2024-04-26T21:33:31.058608+0000","last_became_active":"2024-04-26T21:32:44.862141+0000","last_became_peered":"2024-04-26T21:32:44.862141+0000","last_unstale":"2024-04-26T21:33:31.058608+0000","last_undegraded":"2024-04-26T21:33:31.058608+0000","last_fullsized":"2024-04-26T21:33:31.058608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:32:43.824023+0000","last_clean_scrub_stamp":"2024-04-26T21:32:43.824023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:52:55.919407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280902,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":53869,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":475,"num_read_kb":294,"num_write":157,"num_write_kb":97,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":368640,"data_stored":161607,"data_compressed":5583,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383711,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":456134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645347,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1077248,"data_stored":1097483,"data_compressed":37588,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808872,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":958464,"data_stored":456370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070510,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":958464,"data_stored":456290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332147,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1007616,"data_stored":1048611,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495672,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":946176,"data_stored":455759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574930,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40472,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704680,"statfs":{"total":95995035648,"available":95953592320,"internally_reserved":0,"allocated":991232,"data_stored":505029,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738455,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":457030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411403,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":506658,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182471,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36284,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708868,"statfs":{"total":95995035648,"available":95957880832,"internally_reserved":0,"allocated":962560,"data_stored":456070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920835,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":958464,"data_stored":456699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757310,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1019904,"data_stored":1047904,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34089,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34089,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34089,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35727,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":49477,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":51106,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":51446,"data_compressed":1861,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T21:34:08.259 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-26T21:34:08.259 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-26T21:34:08.259 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-26T21:34:08.259 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-26T21:34:08.514 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:34:08.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:08 smithi050 ceph-mon[32753]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:34:08.882 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:08 smithi046 ceph-mon[29201]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:34:09.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:08 smithi142 ceph-mon[32944]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:34:09.854 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:09 smithi046 ceph-mon[29201]: pgmap v336: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:34:09.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:09 smithi050 ceph-mon[32753]: pgmap v336: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:34:10.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:09 smithi142 ceph-mon[32944]: pgmap v336: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:34:10.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.239+0000 7f83c84b9700 1 -- 172.21.15.46:0/4159911193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83c006bad0 msgr2=0x7f83c01102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:10.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.239+0000 7f83c84b9700 1 --2- 172.21.15.46:0/4159911193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83c006bad0 0x7f83c01102e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f83b800b0a0 tx=0x7f83b800b3b0 comp rx=0 tx=0).stop 2024-04-26T21:34:10.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.239+0000 7f83c84b9700 1 -- 172.21.15.46:0/4159911193 shutdown_connections 2024-04-26T21:34:10.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.239+0000 7f83c84b9700 1 --2- 172.21.15.46:0/4159911193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83c006bad0 0x7f83c01102e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:10.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.239+0000 7f83c84b9700 1 --2- 172.21.15.46:0/4159911193 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83c01094a0 0x7f83c0109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:10.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.239+0000 7f83c84b9700 1 --2- 172.21.15.46:0/4159911193 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83c006b130 0x7f83c006b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:10.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.239+0000 7f83c84b9700 1 -- 172.21.15.46:0/4159911193 >> 172.21.15.46:0/4159911193 conn(0x7f83c00792d0 msgr2=0x7f83c00796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:10.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.239+0000 7f83c84b9700 1 -- 172.21.15.46:0/4159911193 shutdown_connections 2024-04-26T21:34:10.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.239+0000 7f83c84b9700 1 -- 172.21.15.46:0/4159911193 wait complete. 2024-04-26T21:34:10.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.240+0000 7f83c84b9700 1 Processor -- start 2024-04-26T21:34:10.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.240+0000 7f83c84b9700 1 -- start start 2024-04-26T21:34:10.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.240+0000 7f83c84b9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83c006b130 0x7f83c01023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:10.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.240+0000 7f83c84b9700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83c006bad0 0x7f83c0102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:10.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.240+0000 7f83c84b9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83c01094a0 0x7f83c0102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:10.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.240+0000 7f83c84b9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f83c01135a0 con 0x7f83c01094a0 2024-04-26T21:34:10.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.240+0000 7f83c84b9700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f83c0113420 con 0x7f83c006b130 2024-04-26T21:34:10.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.240+0000 7f83c84b9700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f83c0113720 con 0x7f83c006bad0 2024-04-26T21:34:10.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.240+0000 7f83c6a56700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83c01094a0 0x7f83c0102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:10.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.240+0000 7f83c6a56700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83c01094a0 0x7f83c0102e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40156/0 (socket says 172.21.15.46:40156) 2024-04-26T21:34:10.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.240+0000 7f83c6a56700 1 -- 172.21.15.46:0/2099877176 learned_addr learned my addr 172.21.15.46:0/2099877176 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:10.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.240+0000 7f83c6255700 1 --2- 172.21.15.46:0/2099877176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83c006b130 0x7f83c01023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:10.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.240+0000 7f83c5a54700 1 --2- 172.21.15.46:0/2099877176 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83c006bad0 0x7f83c0102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:10.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.240+0000 7f83c6a56700 1 -- 172.21.15.46:0/2099877176 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83c006bad0 msgr2=0x7f83c0102930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:10.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.240+0000 7f83c6a56700 1 --2- 172.21.15.46:0/2099877176 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83c006bad0 0x7f83c0102930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:10.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.240+0000 7f83c6a56700 1 -- 172.21.15.46:0/2099877176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83c006b130 msgr2=0x7f83c01023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:10.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.240+0000 7f83c6a56700 1 --2- 172.21.15.46:0/2099877176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83c006b130 0x7f83c01023f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:10.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.240+0000 7f83c6a56700 1 -- 172.21.15.46:0/2099877176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83c01a7f50 con 0x7f83c01094a0 2024-04-26T21:34:10.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.240+0000 7f83c6a56700 1 --2- 172.21.15.46:0/2099877176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83c01094a0 0x7f83c0102e70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f83b800b830 tx=0x7f83b8004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:10.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.241+0000 7f83b37fe700 1 -- 172.21.15.46:0/2099877176 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b8021070 con 0x7f83c01094a0 2024-04-26T21:34:10.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.241+0000 7f83c84b9700 1 -- 172.21.15.46:0/2099877176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83c01a81e0 con 0x7f83c01094a0 2024-04-26T21:34:10.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.241+0000 7f83b37fe700 1 -- 172.21.15.46:0/2099877176 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83b8007e30 con 0x7f83c01094a0 2024-04-26T21:34:10.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.241+0000 7f83b37fe700 1 -- 172.21.15.46:0/2099877176 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83b80047d0 con 0x7f83c01094a0 2024-04-26T21:34:10.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.241+0000 7f83c84b9700 1 -- 172.21.15.46:0/2099877176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83c01a86e0 con 0x7f83c01094a0 2024-04-26T21:34:10.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.242+0000 7f83c84b9700 1 -- 172.21.15.46:0/2099877176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83c006a110 con 0x7f83c01094a0 2024-04-26T21:34:10.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.243+0000 7f83b37fe700 1 -- 172.21.15.46:0/2099877176 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f83b8007980 con 0x7f83c01094a0 2024-04-26T21:34:10.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.244+0000 7f83b37fe700 1 --2- 172.21.15.46:0/2099877176 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f83ac0a3cd0 0x7f83ac0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:10.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.244+0000 7f83c6255700 1 --2- 172.21.15.46:0/2099877176 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f83ac0a3cd0 0x7f83ac0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:10.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.244+0000 7f83b37fe700 1 -- 172.21.15.46:0/2099877176 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f83b800f020 con 0x7f83c01094a0 2024-04-26T21:34:10.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.244+0000 7f83c6255700 1 --2- 172.21.15.46:0/2099877176 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f83ac0a3cd0 0x7f83ac0a6190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f83bc00abd0 tx=0x7f83bc00a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:10.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.251+0000 7f83b37fe700 1 -- 172.21.15.46:0/2099877176 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f83b802a020 con 0x7f83c01094a0 2024-04-26T21:34:10.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.512+0000 7f83c84b9700 1 -- 172.21.15.46:0/2099877176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f83c002e490 con 0x7f83c01094a0 2024-04-26T21:34:10.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.512+0000 7f83b37fe700 1 -- 172.21.15.46:0/2099877176 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f83b809e820 con 0x7f83c01094a0 2024-04-26T21:34:10.514 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:34:10.514 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-26T21:34:10.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.515+0000 7f83c84b9700 1 -- 172.21.15.46:0/2099877176 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f83ac0a3cd0 msgr2=0x7f83ac0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:10.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.515+0000 7f83c84b9700 1 --2- 172.21.15.46:0/2099877176 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f83ac0a3cd0 0x7f83ac0a6190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f83bc00abd0 tx=0x7f83bc00a380 comp rx=0 tx=0).stop 2024-04-26T21:34:10.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.515+0000 7f83c84b9700 1 -- 172.21.15.46:0/2099877176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83c01094a0 msgr2=0x7f83c0102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:10.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.515+0000 7f83c84b9700 1 --2- 172.21.15.46:0/2099877176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83c01094a0 0x7f83c0102e70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f83b800b830 tx=0x7f83b8004ce0 comp rx=0 tx=0).stop 2024-04-26T21:34:10.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.515+0000 7f83c84b9700 1 -- 172.21.15.46:0/2099877176 shutdown_connections 2024-04-26T21:34:10.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.515+0000 7f83c84b9700 1 --2- 172.21.15.46:0/2099877176 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f83ac0a3cd0 0x7f83ac0a6190 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:10.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.515+0000 7f83c84b9700 1 --2- 172.21.15.46:0/2099877176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83c01094a0 0x7f83c0102e70 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:10.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.515+0000 7f83c84b9700 1 --2- 172.21.15.46:0/2099877176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83c006b130 0x7f83c01023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:10.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.515+0000 7f83c84b9700 1 --2- 172.21.15.46:0/2099877176 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83c006bad0 0x7f83c0102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:10.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.515+0000 7f83c84b9700 1 -- 172.21.15.46:0/2099877176 >> 172.21.15.46:0/2099877176 conn(0x7f83c00792d0 msgr2=0x7f83c0100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:10.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.515+0000 7f83c84b9700 1 -- 172.21.15.46:0/2099877176 shutdown_connections 2024-04-26T21:34:10.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:10.515+0000 7f83c84b9700 1 -- 172.21.15.46:0/2099877176 wait complete. 2024-04-26T21:34:10.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:10 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2099877176' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T21:34:10.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:10 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/2099877176' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T21:34:11.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:10 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/2099877176' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T21:34:11.234 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-26T21:34:11.235 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:34:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.340+0000 7efdb3062640 1 -- 172.21.15.46:0/261540819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdac10d3a0 msgr2=0x7efdac10d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.340+0000 7efdb3062640 1 --2- 172.21.15.46:0/261540819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdac10d3a0 0x7efdac10d7c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7efda0009900 tx=0x7efda002a410 comp rx=0 tx=0).stop 2024-04-26T21:34:11.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.340+0000 7efdb3062640 1 -- 172.21.15.46:0/261540819 shutdown_connections 2024-04-26T21:34:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.340+0000 7efdb3062640 1 --2- 172.21.15.46:0/261540819 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efdac10ef50 0x7efdac1157e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.340+0000 7efdb3062640 1 --2- 172.21.15.46:0/261540819 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdac10e590 0x7efdac10ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.340+0000 7efdb3062640 1 --2- 172.21.15.46:0/261540819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdac10d3a0 0x7efdac10d7c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.340+0000 7efdb3062640 1 -- 172.21.15.46:0/261540819 >> 172.21.15.46:0/261540819 conn(0x7efdac06d400 msgr2=0x7efdac06d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.340+0000 7efdb3062640 1 -- 172.21.15.46:0/261540819 shutdown_connections 2024-04-26T21:34:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.340+0000 7efdb3062640 1 -- 172.21.15.46:0/261540819 wait complete. 2024-04-26T21:34:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.341+0000 7efdb3062640 1 Processor -- start 2024-04-26T21:34:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.341+0000 7efdb3062640 1 -- start start 2024-04-26T21:34:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.341+0000 7efdb3062640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efdac10d3a0 0x7efdac1a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.341+0000 7efdb3062640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdac10e590 0x7efdac1a3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.341+0000 7efdb3062640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdac10ef50 0x7efdac1aabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.341+0000 7efdb3062640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efdac118f30 con 0x7efdac10e590 2024-04-26T21:34:11.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.341+0000 7efdb3062640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7efdac118db0 con 0x7efdac10d3a0 2024-04-26T21:34:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.341+0000 7efdb3062640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efdac1190b0 con 0x7efdac10ef50 2024-04-26T21:34:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.341+0000 7efdabfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdac10e590 0x7efdac1a3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.341+0000 7efdabfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdac10e590 0x7efdac1a3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40172/0 (socket says 172.21.15.46:40172) 2024-04-26T21:34:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.341+0000 7efdabfff640 1 -- 172.21.15.46:0/1161427666 learned_addr learned my addr 172.21.15.46:0/1161427666 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.341+0000 7efdb0dfe640 1 --2- 172.21.15.46:0/1161427666 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efdac10d3a0 0x7efdac1a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.341+0000 7efdb15ff640 1 --2- 172.21.15.46:0/1161427666 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdac10ef50 0x7efdac1aabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.341+0000 7efdabfff640 1 -- 172.21.15.46:0/1161427666 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdac10ef50 msgr2=0x7efdac1aabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.341+0000 7efdabfff640 1 --2- 172.21.15.46:0/1161427666 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdac10ef50 0x7efdac1aabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.341+0000 7efdabfff640 1 -- 172.21.15.46:0/1161427666 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efdac10d3a0 msgr2=0x7efdac1a3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.341+0000 7efdabfff640 1 --2- 172.21.15.46:0/1161427666 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efdac10d3a0 0x7efdac1a3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.341+0000 7efdabfff640 1 -- 172.21.15.46:0/1161427666 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdac1ab230 con 0x7efdac10e590 2024-04-26T21:34:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.342+0000 7efdabfff640 1 --2- 172.21.15.46:0/1161427666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdac10e590 0x7efdac1a3b50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7efd9800d770 tx=0x7efd9800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.342+0000 7efda9ffb640 1 -- 172.21.15.46:0/1161427666 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd98017070 con 0x7efdac10e590 2024-04-26T21:34:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.342+0000 7efdb3062640 1 -- 172.21.15.46:0/1161427666 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdac1ab520 con 0x7efdac10e590 2024-04-26T21:34:11.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.342+0000 7efda9ffb640 1 -- 172.21.15.46:0/1161427666 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efd980054d0 con 0x7efdac10e590 2024-04-26T21:34:11.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.342+0000 7efda9ffb640 1 -- 172.21.15.46:0/1161427666 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd98004c30 con 0x7efdac10e590 2024-04-26T21:34:11.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.342+0000 7efdb3062640 1 -- 172.21.15.46:0/1161427666 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdac1aba20 con 0x7efdac10e590 2024-04-26T21:34:11.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.343+0000 7efdb3062640 1 -- 172.21.15.46:0/1161427666 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdac114790 con 0x7efdac10e590 2024-04-26T21:34:11.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.344+0000 7efda9ffb640 1 -- 172.21.15.46:0/1161427666 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7efd98004e10 con 0x7efdac10e590 2024-04-26T21:34:11.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.344+0000 7efda9ffb640 1 --2- 172.21.15.46:0/1161427666 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7efd9c0a3c40 0x7efd9c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:11.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.345+0000 7efda9ffb640 1 -- 172.21.15.46:0/1161427666 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7efd980d00d0 con 0x7efdac10e590 2024-04-26T21:34:11.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.345+0000 7efdb0dfe640 1 --2- 172.21.15.46:0/1161427666 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7efd9c0a3c40 0x7efd9c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:11.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.345+0000 7efdb0dfe640 1 --2- 172.21.15.46:0/1161427666 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7efd9c0a3c40 0x7efd9c0a6100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7efda0006010 tx=0x7efda00314a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:11.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.349+0000 7efda9ffb640 1 -- 172.21.15.46:0/1161427666 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7efd9809c980 con 0x7efdac10e590 2024-04-26T21:34:11.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.551+0000 7efdb3062640 1 -- 172.21.15.46:0/1161427666 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efdac02dc80 con 0x7efdac10e590 2024-04-26T21:34:11.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.553+0000 7efda9ffb640 1 -- 172.21.15.46:0/1161427666 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22602 (secure 0 0 0) 0x7efd9809c280 con 0x7efdac10e590 2024-04-26T21:34:11.554 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:34:11.554 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":87,"fsid":"36c7013a-0413-11ef-bc93-c7b262605968","created":"2024-04-26T21:24:30.481156+0000","modified":"2024-04-26T21:33:31.051554+0000","last_up_change":"2024-04-26T21:31:50.100491+0000","last_in_change":"2024-04-26T21:31:29.643428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:27:57.935988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:32:42.195211+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:32:43.162114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f53c8702-e4da-435f-a9dc-3911d0d36ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6803","nonce":2762665125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6805","nonce":2762665125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6809","nonce":2762665125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6807","nonce":2762665125}]},"public_addr":"172.21.15.46:6803/2762665125","cluster_addr":"172.21.15.46:6805/2762665125","heartbeat_back_addr":"172.21.15.46:6809/2762665125","heartbeat_front_addr":"172.21.15.46:6807/2762665125","state":["exists","up"]},{"osd":1,"uuid":"fc4d6f52-fc8a-4efd-80c9-c092b57e7f42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6801","nonce":1628008970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6803","nonce":1628008970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6807","nonce":1628008970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6805","nonce":1628008970}]},"public_addr":"172.21.15.50:6801/1628008970","cluster_addr":"172.21.15.50:6803/1628008970","heartbeat_back_addr":"172.21.15.50:6807/1628008970","heartbeat_front_addr":"172.21.15.50:6805/1628008970","state":["exists","up"]},{"osd":2,"uuid":"26d1baa1-4ea1-4b7e-9fcf-4f8bbf7329c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6801","nonce":1513279137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6803","nonce":1513279137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6807","nonce":1513279137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6805","nonce":1513279137}]},"public_addr":"172.21.15.142:6801/1513279137","cluster_addr":"172.21.15.142:6803/1513279137","heartbeat_back_addr":"172.21.15.142:6807/1513279137","heartbeat_front_addr":"172.21.15.142:6805/1513279137","state":["exists","up"]},{"osd":3,"uuid":"34d2401f-5fcc-491c-8fcf-c3c492611f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6811","nonce":2435181760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6813","nonce":2435181760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6817","nonce":2435181760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6815","nonce":2435181760}]},"public_addr":"172.21.15.46:6811/2435181760","cluster_addr":"172.21.15.46:6813/2435181760","heartbeat_back_addr":"172.21.15.46:6817/2435181760","heartbeat_front_addr":"172.21.15.46:6815/2435181760","state":["exists","up"]},{"osd":4,"uuid":"986bb709-e4cf-400d-982e-c3f5ac413e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6809","nonce":3177333250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6811","nonce":3177333250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6815","nonce":3177333250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6813","nonce":3177333250}]},"public_addr":"172.21.15.50:6809/3177333250","cluster_addr":"172.21.15.50:6811/3177333250","heartbeat_back_addr":"172.21.15.50:6815/3177333250","heartbeat_front_addr":"172.21.15.50:6813/3177333250","state":["exists","up"]},{"osd":5,"uuid":"b7d65851-ebe1-45e2-833d-8022446f2698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6809","nonce":4113107737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6811","nonce":4113107737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6815","nonce":4113107737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6813","nonce":4113107737}]},"public_addr":"172.21.15.142:6809/4113107737","cluster_addr":"172.21.15.142:6811/4113107737","heartbeat_back_addr":"172.21.15.142:6815/4113107737","heartbeat_front_addr":"172.21.15.142:6813/4113107737","state":["exists","up"]},{"osd":6,"uuid":"2a3e96d7-d5bf-44ff-9134-29f1420c0e4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6819","nonce":2325162882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6821","nonce":2325162882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6825","nonce":2325162882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6823","nonce":2325162882}]},"public_addr":"172.21.15.46:6819/2325162882","cluster_addr":"172.21.15.46:6821/2325162882","heartbeat_back_addr":"172.21.15.46:6825/2325162882","heartbeat_front_addr":"172.21.15.46:6823/2325162882","state":["exists","up"]},{"osd":7,"uuid":"e53ae885-23b6-46fd-b4b1-c62f31a6921f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6817","nonce":820310228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6819","nonce":820310228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6823","nonce":820310228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6821","nonce":820310228}]},"public_addr":"172.21.15.50:6817/820310228","cluster_addr":"172.21.15.50:6819/820310228","heartbeat_back_addr":"172.21.15.50:6823/820310228","heartbeat_front_addr":"172.21.15.50:6821/820310228","state":["exists","up"]},{"osd":8,"uuid":"356e2692-16a9-4045-b58f-c8f9cb5d7241","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6817","nonce":1824918711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6819","nonce":1824918711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6823","nonce":1824918711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6821","nonce":1824918711}]},"public_addr":"172.21.15.142:6817/1824918711","cluster_addr":"172.21.15.142:6819/1824918711","heartbeat_back_addr":"172.21.15.142:6823/1824918711","heartbeat_front_addr":"172.21.15.142:6821/1824918711","state":["exists","up"]},{"osd":9,"uuid":"8e55570e-77e5-4892-8b9f-77dcd03a4eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6827","nonce":3601947736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6829","nonce":3601947736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6833","nonce":3601947736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6831","nonce":3601947736}]},"public_addr":"172.21.15.46:6827/3601947736","cluster_addr":"172.21.15.46:6829/3601947736","heartbeat_back_addr":"172.21.15.46:6833/3601947736","heartbeat_front_addr":"172.21.15.46:6831/3601947736","state":["exists","up"]},{"osd":10,"uuid":"366389fa-fb9a-408b-b5c2-343c054c5a64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6825","nonce":3135765006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6827","nonce":3135765006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6831","nonce":3135765006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6829","nonce":3135765006}]},"public_addr":"172.21.15.50:6825/3135765006","cluster_addr":"172.21.15.50:6827/3135765006","heartbeat_back_addr":"172.21.15.50:6831/3135765006","heartbeat_front_addr":"172.21.15.50:6829/3135765006","state":["exists","up"]},{"osd":11,"uuid":"7b6e3d93-59e1-4853-b4a2-261b728c47a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6825","nonce":1482712612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6827","nonce":1482712612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6831","nonce":1482712612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6829","nonce":1482712612}]},"public_addr":"172.21.15.142:6825/1482712612","cluster_addr":"172.21.15.142:6827/1482712612","heartbeat_back_addr":"172.21.15.142:6831/1482712612","heartbeat_front_addr":"172.21.15.142:6829/1482712612","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:08.082015+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:32.634928+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:55.978864+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:21.796863+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:46.862518+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:11.440432+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:37.764777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:03.374958+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:28.714052+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:55.761167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:22.330951+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:47.839334+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/2465242088":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6801/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6800/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/1381997090":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/3507268217":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2570024356":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/2691061938":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2775954801":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6800/2896098005":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6801/2896098005":"2024-04-27T21:25:02.078730+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:34:11.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.555+0000 7efdb3062640 1 -- 172.21.15.46:0/1161427666 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7efd9c0a3c40 msgr2=0x7efd9c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:11.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.555+0000 7efdb3062640 1 --2- 172.21.15.46:0/1161427666 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7efd9c0a3c40 0x7efd9c0a6100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7efda0006010 tx=0x7efda00314a0 comp rx=0 tx=0).stop 2024-04-26T21:34:11.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.555+0000 7efdb3062640 1 -- 172.21.15.46:0/1161427666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdac10e590 msgr2=0x7efdac1a3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:11.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.555+0000 7efdb3062640 1 --2- 172.21.15.46:0/1161427666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdac10e590 0x7efdac1a3b50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7efd9800d770 tx=0x7efd9800dc30 comp rx=0 tx=0).stop 2024-04-26T21:34:11.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.555+0000 7efdb3062640 1 -- 172.21.15.46:0/1161427666 shutdown_connections 2024-04-26T21:34:11.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.555+0000 7efdb3062640 1 --2- 172.21.15.46:0/1161427666 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7efd9c0a3c40 0x7efd9c0a6100 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:11.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.555+0000 7efdb3062640 1 --2- 172.21.15.46:0/1161427666 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efdac10ef50 0x7efdac1aabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:11.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.555+0000 7efdb3062640 1 --2- 172.21.15.46:0/1161427666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efdac10e590 0x7efdac1a3b50 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:11.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.555+0000 7efdb3062640 1 --2- 172.21.15.46:0/1161427666 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efdac10d3a0 0x7efdac1a3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:11.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.555+0000 7efdb3062640 1 -- 172.21.15.46:0/1161427666 >> 172.21.15.46:0/1161427666 conn(0x7efdac06d400 msgr2=0x7efdac111960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:11.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.555+0000 7efdb3062640 1 -- 172.21.15.46:0/1161427666 shutdown_connections 2024-04-26T21:34:11.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.555+0000 7efdb3062640 1 -- 172.21.15.46:0/1161427666 wait complete. 2024-04-26T21:34:11.567 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T21:27:57.935988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T21:32:42.195211+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T21:32:43.162114+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T21:34:11.568 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T21:34:11.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.672+0000 7f583d6f8640 1 -- 172.21.15.46:0/3725443707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5838103410 msgr2=0x7f5838105800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:11.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.672+0000 7f583d6f8640 1 --2- 172.21.15.46:0/3725443707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5838103410 0x7f5838105800 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5828009900 tx=0x7f582802a410 comp rx=0 tx=0).stop 2024-04-26T21:34:11.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.672+0000 7f583d6f8640 1 -- 172.21.15.46:0/3725443707 shutdown_connections 2024-04-26T21:34:11.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.672+0000 7f583d6f8640 1 --2- 172.21.15.46:0/3725443707 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5838108670 0x7f583810aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:11.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.672+0000 7f583d6f8640 1 --2- 172.21.15.46:0/3725443707 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5838105d40 0x7f5838108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:11.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.672+0000 7f583d6f8640 1 --2- 172.21.15.46:0/3725443707 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5838103410 0x7f5838105800 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:11.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.672+0000 7f583d6f8640 1 -- 172.21.15.46:0/3725443707 >> 172.21.15.46:0/3725443707 conn(0x7f58380fce20 msgr2=0x7f58380ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:11.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.672+0000 7f583d6f8640 1 -- 172.21.15.46:0/3725443707 shutdown_connections 2024-04-26T21:34:11.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.672+0000 7f583d6f8640 1 -- 172.21.15.46:0/3725443707 wait complete. 2024-04-26T21:34:11.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.672+0000 7f583d6f8640 1 Processor -- start 2024-04-26T21:34:11.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.673+0000 7f583d6f8640 1 -- start start 2024-04-26T21:34:11.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.673+0000 7f583d6f8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5838103410 0x7f5838198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:11.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.673+0000 7f583d6f8640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5838105d40 0x7f5838199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:11.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.673+0000 7f583d6f8640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5838108670 0x7f58381a0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:11.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.673+0000 7f583d6f8640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f583810e4c0 con 0x7f5838103410 2024-04-26T21:34:11.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.673+0000 7f583d6f8640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f583810e340 con 0x7f5838108670 2024-04-26T21:34:11.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.673+0000 7f583d6f8640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f583810e640 con 0x7f5838105d40 2024-04-26T21:34:11.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.673+0000 7f5836ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5838103410 0x7f5838198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:11.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.673+0000 7f5836ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5838103410 0x7f5838198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40192/0 (socket says 172.21.15.46:40192) 2024-04-26T21:34:11.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.673+0000 7f5836ffd640 1 -- 172.21.15.46:0/77273598 learned_addr learned my addr 172.21.15.46:0/77273598 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:11.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.673+0000 7f58377fe640 1 --2- 172.21.15.46:0/77273598 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5838108670 0x7f58381a0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:11.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.673+0000 7f58367fc640 1 --2- 172.21.15.46:0/77273598 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5838105d40 0x7f5838199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:11.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.673+0000 7f5836ffd640 1 -- 172.21.15.46:0/77273598 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5838105d40 msgr2=0x7f5838199100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:11.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.673+0000 7f5836ffd640 1 --2- 172.21.15.46:0/77273598 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5838105d40 0x7f5838199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:11.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.673+0000 7f5836ffd640 1 -- 172.21.15.46:0/77273598 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5838108670 msgr2=0x7f58381a0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:11.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.673+0000 7f5836ffd640 1 --2- 172.21.15.46:0/77273598 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5838108670 0x7f58381a0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:11.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.673+0000 7f5836ffd640 1 -- 172.21.15.46:0/77273598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58381a07e0 con 0x7f5838103410 2024-04-26T21:34:11.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.673+0000 7f5836ffd640 1 --2- 172.21.15.46:0/77273598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5838103410 0x7f5838198bc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5828005940 tx=0x7f5828004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:11.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.673+0000 7f581ffff640 1 -- 172.21.15.46:0/77273598 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f582803f030 con 0x7f5838103410 2024-04-26T21:34:11.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.674+0000 7f583d6f8640 1 -- 172.21.15.46:0/77273598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58381a0a70 con 0x7f5838103410 2024-04-26T21:34:11.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.674+0000 7f581ffff640 1 -- 172.21.15.46:0/77273598 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5828003cb0 con 0x7f5838103410 2024-04-26T21:34:11.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.674+0000 7f581ffff640 1 -- 172.21.15.46:0/77273598 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5828035070 con 0x7f5838103410 2024-04-26T21:34:11.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.674+0000 7f583d6f8640 1 -- 172.21.15.46:0/77273598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58381a0f60 con 0x7f5838103410 2024-04-26T21:34:11.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.675+0000 7f583d6f8640 1 -- 172.21.15.46:0/77273598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58380fe910 con 0x7f5838103410 2024-04-26T21:34:11.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.676+0000 7f581ffff640 1 -- 172.21.15.46:0/77273598 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5828042030 con 0x7f5838103410 2024-04-26T21:34:11.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.676+0000 7f581ffff640 1 --2- 172.21.15.46:0/77273598 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f58200a3c40 0x7f58200a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:11.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.676+0000 7f58367fc640 1 --2- 172.21.15.46:0/77273598 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f58200a3c40 0x7f58200a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:11.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.676+0000 7f581ffff640 1 -- 172.21.15.46:0/77273598 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f58280f1130 con 0x7f5838103410 2024-04-26T21:34:11.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.677+0000 7f58367fc640 1 --2- 172.21.15.46:0/77273598 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f58200a3c40 0x7f58200a6100 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f583819a0d0 tx=0x7f582c005cc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:11.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.680+0000 7f581ffff640 1 -- 172.21.15.46:0/77273598 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f58280bd9e0 con 0x7f5838103410 2024-04-26T21:34:11.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.866+0000 7f583d6f8640 1 -- 172.21.15.46:0/77273598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f583802dd40 con 0x7f5838103410 2024-04-26T21:34:11.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.867+0000 7f581ffff640 1 -- 172.21.15.46:0/77273598 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f5828003e60 con 0x7f5838103410 2024-04-26T21:34:11.868 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-26T21:34:11.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:11 smithi050 ceph-mon[32753]: pgmap v337: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T21:34:11.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:11 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/1161427666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:34:11.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.869+0000 7f583d6f8640 1 -- 172.21.15.46:0/77273598 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f58200a3c40 msgr2=0x7f58200a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:11.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.869+0000 7f583d6f8640 1 --2- 172.21.15.46:0/77273598 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f58200a3c40 0x7f58200a6100 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f583819a0d0 tx=0x7f582c005cc0 comp rx=0 tx=0).stop 2024-04-26T21:34:11.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.869+0000 7f583d6f8640 1 -- 172.21.15.46:0/77273598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5838103410 msgr2=0x7f5838198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:11.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.869+0000 7f583d6f8640 1 --2- 172.21.15.46:0/77273598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5838103410 0x7f5838198bc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5828005940 tx=0x7f5828004430 comp rx=0 tx=0).stop 2024-04-26T21:34:11.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.869+0000 7f583d6f8640 1 -- 172.21.15.46:0/77273598 shutdown_connections 2024-04-26T21:34:11.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.869+0000 7f583d6f8640 1 --2- 172.21.15.46:0/77273598 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f58200a3c40 0x7f58200a6100 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:11.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.869+0000 7f583d6f8640 1 --2- 172.21.15.46:0/77273598 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5838108670 0x7f58381a0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:11.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.869+0000 7f583d6f8640 1 --2- 172.21.15.46:0/77273598 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5838105d40 0x7f5838199100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:11.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.869+0000 7f583d6f8640 1 --2- 172.21.15.46:0/77273598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5838103410 0x7f5838198bc0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:11.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.869+0000 7f583d6f8640 1 -- 172.21.15.46:0/77273598 >> 172.21.15.46:0/77273598 conn(0x7f58380fce20 msgr2=0x7f5838103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:11.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.869+0000 7f583d6f8640 1 -- 172.21.15.46:0/77273598 shutdown_connections 2024-04-26T21:34:11.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.869+0000 7f583d6f8640 1 -- 172.21.15.46:0/77273598 wait complete. 2024-04-26T21:34:11.880 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:11 smithi046 ceph-mon[29201]: pgmap v337: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T21:34:11.880 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:11 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1161427666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:34:11.881 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T21:34:11.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.985+0000 7f013f19d640 1 -- 172.21.15.46:0/877677632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0138105d40 msgr2=0x7f0138108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:11.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.985+0000 7f013f19d640 1 --2- 172.21.15.46:0/877677632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0138105d40 0x7f0138108130 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0124009900 tx=0x7f012402a410 comp rx=0 tx=0).stop 2024-04-26T21:34:11.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.985+0000 7f013f19d640 1 -- 172.21.15.46:0/877677632 shutdown_connections 2024-04-26T21:34:11.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.985+0000 7f013f19d640 1 --2- 172.21.15.46:0/877677632 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0138108670 0x7f013810aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:11.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.985+0000 7f013f19d640 1 --2- 172.21.15.46:0/877677632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0138105d40 0x7f0138108130 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:11.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.985+0000 7f013f19d640 1 --2- 172.21.15.46:0/877677632 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0138103410 0x7f0138105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.985+0000 7f013f19d640 1 -- 172.21.15.46:0/877677632 >> 172.21.15.46:0/877677632 conn(0x7f01380fce20 msgr2=0x7f01380ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.985+0000 7f013f19d640 1 -- 172.21.15.46:0/877677632 shutdown_connections 2024-04-26T21:34:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.985+0000 7f013f19d640 1 -- 172.21.15.46:0/877677632 wait complete. 2024-04-26T21:34:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.985+0000 7f013f19d640 1 Processor -- start 2024-04-26T21:34:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.986+0000 7f013f19d640 1 -- start start 2024-04-26T21:34:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.986+0000 7f013f19d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0138103410 0x7f0138102d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.986+0000 7f013f19d640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0138105d40 0x7f01381013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.986+0000 7f013f19d640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0138108670 0x7f0138101900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.986+0000 7f013f19d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f013810e4b0 con 0x7f0138103410 2024-04-26T21:34:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.986+0000 7f013f19d640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f013810e330 con 0x7f0138108670 2024-04-26T21:34:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.986+0000 7f013f19d640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f013810e630 con 0x7f0138105d40 2024-04-26T21:34:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.986+0000 7f013cf39640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0138103410 0x7f0138102d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.986+0000 7f013cf39640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0138103410 0x7f0138102d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40210/0 (socket says 172.21.15.46:40210) 2024-04-26T21:34:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.986+0000 7f013cf39640 1 -- 172.21.15.46:0/1286504339 learned_addr learned my addr 172.21.15.46:0/1286504339 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:11.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.986+0000 7f0137fff640 1 --2- 172.21.15.46:0/1286504339 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0138105d40 0x7f01381013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.986+0000 7f013cf39640 1 -- 172.21.15.46:0/1286504339 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0138105d40 msgr2=0x7f01381013c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.986+0000 7f013cf39640 1 --2- 172.21.15.46:0/1286504339 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0138105d40 0x7f01381013c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.986+0000 7f013cf39640 1 -- 172.21.15.46:0/1286504339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0138108670 msgr2=0x7f0138101900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.986+0000 7f013d73a640 1 --2- 172.21.15.46:0/1286504339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0138108670 0x7f0138101900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.986+0000 7f013cf39640 1 --2- 172.21.15.46:0/1286504339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0138108670 0x7f0138101900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.986+0000 7f013cf39640 1 -- 172.21.15.46:0/1286504339 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0138102190 con 0x7f0138103410 2024-04-26T21:34:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.986+0000 7f0137fff640 1 --2- 172.21.15.46:0/1286504339 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0138105d40 0x7f01381013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:34:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.986+0000 7f013cf39640 1 --2- 172.21.15.46:0/1286504339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0138103410 0x7f0138102d80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f012c00d950 tx=0x7f012c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.986+0000 7f0135ffb640 1 -- 172.21.15.46:0/1286504339 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012c017070 con 0x7f0138103410 2024-04-26T21:34:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.986+0000 7f0135ffb640 1 -- 172.21.15.46:0/1286504339 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f012c009c30 con 0x7f0138103410 2024-04-26T21:34:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.986+0000 7f0135ffb640 1 -- 172.21.15.46:0/1286504339 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012c00f620 con 0x7f0138103410 2024-04-26T21:34:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.986+0000 7f013f19d640 1 -- 172.21.15.46:0/1286504339 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01381a2cb0 con 0x7f0138103410 2024-04-26T21:34:11.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.987+0000 7f013f19d640 1 -- 172.21.15.46:0/1286504339 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01381a31b0 con 0x7f0138103410 2024-04-26T21:34:11.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.988+0000 7f013f19d640 1 -- 172.21.15.46:0/1286504339 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01380fe910 con 0x7f0138103410 2024-04-26T21:34:11.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.989+0000 7f0135ffb640 1 -- 172.21.15.46:0/1286504339 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f012c010460 con 0x7f0138103410 2024-04-26T21:34:11.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.989+0000 7f0135ffb640 1 --2- 172.21.15.46:0/1286504339 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f01280a3c40 0x7f01280a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:11.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.989+0000 7f0137fff640 1 --2- 172.21.15.46:0/1286504339 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f01280a3c40 0x7f01280a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:11.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.989+0000 7f0135ffb640 1 -- 172.21.15.46:0/1286504339 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f012c0d0240 con 0x7f0138103410 2024-04-26T21:34:11.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.990+0000 7f0137fff640 1 --2- 172.21.15.46:0/1286504339 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f01280a3c40 0x7f01280a6100 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0124006010 tx=0x7f0124000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:11.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:11.994+0000 7f0135ffb640 1 -- 172.21.15.46:0/1286504339 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f012c09caf0 con 0x7f0138103410 2024-04-26T21:34:12.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:11 smithi142 ceph-mon[32944]: pgmap v337: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T21:34:12.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:11 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/1161427666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:34:12.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.179+0000 7f013f19d640 1 -- 172.21.15.46:0/1286504339 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f013802dd80 con 0x7f0138103410 2024-04-26T21:34:12.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.180+0000 7f0135ffb640 1 -- 172.21.15.46:0/1286504339 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f012c009de0 con 0x7f0138103410 2024-04-26T21:34:12.181 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-26T21:34:12.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.182+0000 7f013f19d640 1 -- 172.21.15.46:0/1286504339 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f01280a3c40 msgr2=0x7f01280a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:12.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.182+0000 7f013f19d640 1 --2- 172.21.15.46:0/1286504339 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f01280a3c40 0x7f01280a6100 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0124006010 tx=0x7f0124000bc0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.182+0000 7f013f19d640 1 -- 172.21.15.46:0/1286504339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0138103410 msgr2=0x7f0138102d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:12.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.182+0000 7f013f19d640 1 --2- 172.21.15.46:0/1286504339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0138103410 0x7f0138102d80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f012c00d950 tx=0x7f012c00de10 comp rx=0 tx=0).stop 2024-04-26T21:34:12.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.182+0000 7f013f19d640 1 -- 172.21.15.46:0/1286504339 shutdown_connections 2024-04-26T21:34:12.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.182+0000 7f013f19d640 1 --2- 172.21.15.46:0/1286504339 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f01280a3c40 0x7f01280a6100 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.182+0000 7f013f19d640 1 --2- 172.21.15.46:0/1286504339 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0138108670 0x7f0138101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.182+0000 7f013f19d640 1 --2- 172.21.15.46:0/1286504339 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0138105d40 0x7f01381013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.182+0000 7f013f19d640 1 --2- 172.21.15.46:0/1286504339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0138103410 0x7f0138102d80 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.182+0000 7f013f19d640 1 -- 172.21.15.46:0/1286504339 >> 172.21.15.46:0/1286504339 conn(0x7f01380fce20 msgr2=0x7f0138103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:12.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.182+0000 7f013f19d640 1 -- 172.21.15.46:0/1286504339 shutdown_connections 2024-04-26T21:34:12.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.182+0000 7f013f19d640 1 -- 172.21.15.46:0/1286504339 wait complete. 2024-04-26T21:34:12.194 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T21:34:12.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.297+0000 7fc4f7cda640 1 -- 172.21.15.46:0/581882322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4f0102990 msgr2=0x7fc4f0102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:12.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.297+0000 7fc4f7cda640 1 --2- 172.21.15.46:0/581882322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4f0102990 0x7fc4f0102db0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec009900 tx=0x7fc4ec02a410 comp rx=0 tx=0).stop 2024-04-26T21:34:12.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.297+0000 7fc4f7cda640 1 -- 172.21.15.46:0/581882322 shutdown_connections 2024-04-26T21:34:12.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.297+0000 7fc4f7cda640 1 --2- 172.21.15.46:0/581882322 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4f0104540 0x7fc4f010add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.297+0000 7fc4f7cda640 1 --2- 172.21.15.46:0/581882322 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4f0103b80 0x7fc4f0104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.297+0000 7fc4f7cda640 1 --2- 172.21.15.46:0/581882322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4f0102990 0x7fc4f0102db0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.297+0000 7fc4f7cda640 1 -- 172.21.15.46:0/581882322 >> 172.21.15.46:0/581882322 conn(0x7fc4f00fdf00 msgr2=0x7fc4f0100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:12.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.297+0000 7fc4f7cda640 1 -- 172.21.15.46:0/581882322 shutdown_connections 2024-04-26T21:34:12.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.297+0000 7fc4f7cda640 1 -- 172.21.15.46:0/581882322 wait complete. 2024-04-26T21:34:12.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.298+0000 7fc4f7cda640 1 Processor -- start 2024-04-26T21:34:12.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.298+0000 7fc4f7cda640 1 -- start start 2024-04-26T21:34:12.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.298+0000 7fc4f7cda640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4f0102990 0x7fc4f0198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:12.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.298+0000 7fc4f7cda640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4f0103b80 0x7fc4f0199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:12.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.298+0000 7fc4f7cda640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4f0104540 0x7fc4f01a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:12.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.298+0000 7fc4f7cda640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f010e490 con 0x7fc4f0104540 2024-04-26T21:34:12.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.298+0000 7fc4f7cda640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f010e310 con 0x7fc4f0103b80 2024-04-26T21:34:12.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.298+0000 7fc4f7cda640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f010e610 con 0x7fc4f0102990 2024-04-26T21:34:12.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.298+0000 7fc4f6277640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4f0104540 0x7fc4f01a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:12.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.298+0000 7fc4f6277640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4f0104540 0x7fc4f01a0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40222/0 (socket says 172.21.15.46:40222) 2024-04-26T21:34:12.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.298+0000 7fc4f6277640 1 -- 172.21.15.46:0/511565703 learned_addr learned my addr 172.21.15.46:0/511565703 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:12.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.298+0000 7fc4f5275640 1 --2- 172.21.15.46:0/511565703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4f0103b80 0x7fc4f0199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:12.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.298+0000 7fc4f5a76640 1 --2- 172.21.15.46:0/511565703 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4f0102990 0x7fc4f0198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:12.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.298+0000 7fc4f6277640 1 -- 172.21.15.46:0/511565703 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4f0102990 msgr2=0x7fc4f0198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:12.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.298+0000 7fc4f6277640 1 --2- 172.21.15.46:0/511565703 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4f0102990 0x7fc4f0198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.298+0000 7fc4f6277640 1 -- 172.21.15.46:0/511565703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4f0103b80 msgr2=0x7fc4f0199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:12.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.298+0000 7fc4f6277640 1 --2- 172.21.15.46:0/511565703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4f0103b80 0x7fc4f0199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.298+0000 7fc4f6277640 1 -- 172.21.15.46:0/511565703 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4f01a07f0 con 0x7fc4f0104540 2024-04-26T21:34:12.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.299+0000 7fc4f6277640 1 --2- 172.21.15.46:0/511565703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4f0104540 0x7fc4f01a0180 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc4e800eb10 tx=0x7fc4e800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:12.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.299+0000 7fc4e6ffd640 1 -- 172.21.15.46:0/511565703 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8004b20 con 0x7fc4f0104540 2024-04-26T21:34:12.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.299+0000 7fc4f7cda640 1 -- 172.21.15.46:0/511565703 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4f01a0ae0 con 0x7fc4f0104540 2024-04-26T21:34:12.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.299+0000 7fc4e6ffd640 1 -- 172.21.15.46:0/511565703 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4e8004cc0 con 0x7fc4f0104540 2024-04-26T21:34:12.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.299+0000 7fc4e6ffd640 1 -- 172.21.15.46:0/511565703 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8010670 con 0x7fc4f0104540 2024-04-26T21:34:12.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.299+0000 7fc4f7cda640 1 -- 172.21.15.46:0/511565703 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4f01a1030 con 0x7fc4f0104540 2024-04-26T21:34:12.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.300+0000 7fc4f7cda640 1 -- 172.21.15.46:0/511565703 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4f0109dd0 con 0x7fc4f0104540 2024-04-26T21:34:12.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.301+0000 7fc4e6ffd640 1 -- 172.21.15.46:0/511565703 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc4e8005020 con 0x7fc4f0104540 2024-04-26T21:34:12.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.301+0000 7fc4e6ffd640 1 --2- 172.21.15.46:0/511565703 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fc4dc0a3c40 0x7fc4dc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:12.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.301+0000 7fc4f5a76640 1 --2- 172.21.15.46:0/511565703 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fc4dc0a3c40 0x7fc4dc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:12.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.302+0000 7fc4e6ffd640 1 -- 172.21.15.46:0/511565703 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7fc4e80d0ac0 con 0x7fc4f0104540 2024-04-26T21:34:12.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.302+0000 7fc4f5a76640 1 --2- 172.21.15.46:0/511565703 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fc4dc0a3c40 0x7fc4dc0a6100 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec009230 tx=0x7fc4ec0314a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:12.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.306+0000 7fc4e6ffd640 1 -- 172.21.15.46:0/511565703 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc4e809d370 con 0x7fc4f0104540 2024-04-26T21:34:12.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.494+0000 7fc4f7cda640 1 -- 172.21.15.46:0/511565703 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc4f002dc80 con 0x7fc4f0104540 2024-04-26T21:34:12.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.495+0000 7fc4e6ffd640 1 -- 172.21.15.46:0/511565703 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7fc4e809cc70 con 0x7fc4f0104540 2024-04-26T21:34:12.496 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-26T21:34:12.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.497+0000 7fc4f7cda640 1 -- 172.21.15.46:0/511565703 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fc4dc0a3c40 msgr2=0x7fc4dc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:12.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.497+0000 7fc4f7cda640 1 --2- 172.21.15.46:0/511565703 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fc4dc0a3c40 0x7fc4dc0a6100 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec009230 tx=0x7fc4ec0314a0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.497+0000 7fc4f7cda640 1 -- 172.21.15.46:0/511565703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4f0104540 msgr2=0x7fc4f01a0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:12.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.497+0000 7fc4f7cda640 1 --2- 172.21.15.46:0/511565703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4f0104540 0x7fc4f01a0180 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc4e800eb10 tx=0x7fc4e800c2d0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.497+0000 7fc4f7cda640 1 -- 172.21.15.46:0/511565703 shutdown_connections 2024-04-26T21:34:12.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.497+0000 7fc4f7cda640 1 --2- 172.21.15.46:0/511565703 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fc4dc0a3c40 0x7fc4dc0a6100 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.497+0000 7fc4f7cda640 1 --2- 172.21.15.46:0/511565703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4f0104540 0x7fc4f01a0180 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.497+0000 7fc4f7cda640 1 --2- 172.21.15.46:0/511565703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc4f0103b80 0x7fc4f0199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.497+0000 7fc4f7cda640 1 --2- 172.21.15.46:0/511565703 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc4f0102990 0x7fc4f0198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.497+0000 7fc4f7cda640 1 -- 172.21.15.46:0/511565703 >> 172.21.15.46:0/511565703 conn(0x7fc4f00fdf00 msgr2=0x7fc4f0106fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:12.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.497+0000 7fc4f7cda640 1 -- 172.21.15.46:0/511565703 shutdown_connections 2024-04-26T21:34:12.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.497+0000 7fc4f7cda640 1 -- 172.21.15.46:0/511565703 wait complete. 2024-04-26T21:34:12.509 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:34:12.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.610+0000 7f7ba72d3640 1 -- 172.21.15.46:0/1960929465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ba0105d60 msgr2=0x7f7ba0108150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:12.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.610+0000 7f7ba72d3640 1 --2- 172.21.15.46:0/1960929465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ba0105d60 0x7f7ba0108150 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7b90009900 tx=0x7f7b9002a410 comp rx=0 tx=0).stop 2024-04-26T21:34:12.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.610+0000 7f7ba72d3640 1 -- 172.21.15.46:0/1960929465 shutdown_connections 2024-04-26T21:34:12.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.610+0000 7f7ba72d3640 1 --2- 172.21.15.46:0/1960929465 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ba0108690 0x7f7ba010aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.610+0000 7f7ba72d3640 1 --2- 172.21.15.46:0/1960929465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ba0105d60 0x7f7ba0108150 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.610+0000 7f7ba72d3640 1 --2- 172.21.15.46:0/1960929465 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ba0103430 0x7f7ba0105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.610+0000 7f7ba72d3640 1 -- 172.21.15.46:0/1960929465 >> 172.21.15.46:0/1960929465 conn(0x7f7ba00fce20 msgr2=0x7f7ba00ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:12.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.610+0000 7f7ba72d3640 1 -- 172.21.15.46:0/1960929465 shutdown_connections 2024-04-26T21:34:12.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.610+0000 7f7ba72d3640 1 -- 172.21.15.46:0/1960929465 wait complete. 2024-04-26T21:34:12.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.611+0000 7f7ba72d3640 1 Processor -- start 2024-04-26T21:34:12.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.611+0000 7f7ba72d3640 1 -- start start 2024-04-26T21:34:12.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.611+0000 7f7ba72d3640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ba0103430 0x7f7ba0102da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:12.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.611+0000 7f7ba72d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ba0105d60 0x7f7ba01013e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:12.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.611+0000 7f7ba72d3640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ba0108690 0x7f7ba0101920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:12.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.611+0000 7f7ba72d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba010e590 con 0x7f7ba0105d60 2024-04-26T21:34:12.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.611+0000 7f7ba72d3640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba010e410 con 0x7f7ba0108690 2024-04-26T21:34:12.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.611+0000 7f7ba72d3640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba010e710 con 0x7f7ba0103430 2024-04-26T21:34:12.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.611+0000 7f7ba486e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ba0105d60 0x7f7ba01013e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:12.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.611+0000 7f7ba486e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ba0105d60 0x7f7ba01013e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40234/0 (socket says 172.21.15.46:40234) 2024-04-26T21:34:12.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.611+0000 7f7ba486e640 1 -- 172.21.15.46:0/3488768923 learned_addr learned my addr 172.21.15.46:0/3488768923 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.611+0000 7f7ba506f640 1 --2- 172.21.15.46:0/3488768923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ba0103430 0x7f7ba0102da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.611+0000 7f7ba5870640 1 --2- 172.21.15.46:0/3488768923 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ba0108690 0x7f7ba0101920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.611+0000 7f7ba486e640 1 -- 172.21.15.46:0/3488768923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ba0103430 msgr2=0x7f7ba0102da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.611+0000 7f7ba486e640 1 --2- 172.21.15.46:0/3488768923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ba0103430 0x7f7ba0102da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.611+0000 7f7ba486e640 1 -- 172.21.15.46:0/3488768923 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ba0108690 msgr2=0x7f7ba0101920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.611+0000 7f7ba486e640 1 --2- 172.21.15.46:0/3488768923 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ba0108690 0x7f7ba0101920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.611+0000 7f7ba486e640 1 -- 172.21.15.46:0/3488768923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ba01021b0 con 0x7f7ba0105d60 2024-04-26T21:34:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.611+0000 7f7ba5870640 1 --2- 172.21.15.46:0/3488768923 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ba0108690 0x7f7ba0101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:34:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.611+0000 7f7ba486e640 1 --2- 172.21.15.46:0/3488768923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ba0105d60 0x7f7ba01013e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7b90031500 tx=0x7f7b90003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.612+0000 7f7b967fc640 1 -- 172.21.15.46:0/3488768923 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b9003f070 con 0x7f7ba0105d60 2024-04-26T21:34:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.612+0000 7f7ba72d3640 1 -- 172.21.15.46:0/3488768923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ba01a2d80 con 0x7f7ba0105d60 2024-04-26T21:34:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.612+0000 7f7b967fc640 1 -- 172.21.15.46:0/3488768923 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b9003b460 con 0x7f7ba0105d60 2024-04-26T21:34:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.612+0000 7f7b967fc640 1 -- 172.21.15.46:0/3488768923 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b90035030 con 0x7f7ba0105d60 2024-04-26T21:34:12.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.612+0000 7f7ba72d3640 1 -- 172.21.15.46:0/3488768923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ba01a3290 con 0x7f7ba0105d60 2024-04-26T21:34:12.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.613+0000 7f7ba72d3640 1 -- 172.21.15.46:0/3488768923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ba00fe910 con 0x7f7ba0105d60 2024-04-26T21:34:12.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.614+0000 7f7b967fc640 1 -- 172.21.15.46:0/3488768923 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7b90042070 con 0x7f7ba0105d60 2024-04-26T21:34:12.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.614+0000 7f7b967fc640 1 --2- 172.21.15.46:0/3488768923 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7b8c0a3c40 0x7f7b8c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:12.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.615+0000 7f7ba506f640 1 --2- 172.21.15.46:0/3488768923 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7b8c0a3c40 0x7f7b8c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:12.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.615+0000 7f7b967fc640 1 -- 172.21.15.46:0/3488768923 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f7b900f0f40 con 0x7f7ba0105d60 2024-04-26T21:34:12.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.615+0000 7f7ba506f640 1 --2- 172.21.15.46:0/3488768923 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7b8c0a3c40 0x7f7b8c0a6100 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c009890 tx=0x7f7b9c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:12.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.619+0000 7f7b967fc640 1 -- 172.21.15.46:0/3488768923 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7b900bd7f0 con 0x7f7ba0105d60 2024-04-26T21:34:12.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:12 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/77273598' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:34:12.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:12 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1286504339' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:34:12.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:12 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/511565703' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:34:12.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.844+0000 7f7ba72d3640 1 -- 172.21.15.46:0/3488768923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ba002dad0 con 0x7f7ba0105d60 2024-04-26T21:34:12.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.845+0000 7f7b967fc640 1 -- 172.21.15.46:0/3488768923 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11863 (secure 0 0 0) 0x7f7b900bd0f0 con 0x7f7ba0105d60 2024-04-26T21:34:12.846 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:34:12.846 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3674296286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3674296286},{"type":"v1","addr":"172.21.15.46:6835","nonce":3674296286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/391791206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":391791206},{"type":"v1","addr":"172.21.15.46:6837","nonce":391791206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/1222489912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1222489912},{"type":"v1","addr":"172.21.15.46:6839","nonce":1222489912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14658,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/860605548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":860605548},{"type":"v1","addr":"172.21.15.46:6841","nonce":860605548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3621399933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3621399933},{"type":"v1","addr":"172.21.15.142:6833","nonce":3621399933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/345015778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":345015778},{"type":"v1","addr":"172.21.15.50:6833","nonce":345015778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T21:32:44.201541+0000","modified":"2024-04-26T21:32:49.889029+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24526,"mds_2":24509},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/2546509863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":2546509863},{"type":"v1","addr":"172.21.15.50:6835","nonce":2546509863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/3791468066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":3791468066},{"type":"v1","addr":"172.21.15.142:6835","nonce":3791468066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6837/1723132553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1723132553},{"type":"v1","addr":"172.21.15.142:6837","nonce":1723132553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6837/3189620395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3189620395},{"type":"v1","addr":"172.21.15.50:6837","nonce":3189620395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24526":{"gid":24526,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.142:6839/3089509101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3089509101},{"type":"v1","addr":"172.21.15.142:6839","nonce":3089509101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6839/2453253368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2453253368},{"type":"v1","addr":"172.21.15.50:6839","nonce":2453253368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24509,24526,24539]},"id":1}]} 2024-04-26T21:34:12.846 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 22 2024-04-26T21:34:12.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.847+0000 7f7ba72d3640 1 -- 172.21.15.46:0/3488768923 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7b8c0a3c40 msgr2=0x7f7b8c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:12.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.847+0000 7f7ba72d3640 1 --2- 172.21.15.46:0/3488768923 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7b8c0a3c40 0x7f7b8c0a6100 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c009890 tx=0x7f7b9c008040 comp rx=0 tx=0).stop 2024-04-26T21:34:12.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.847+0000 7f7ba72d3640 1 -- 172.21.15.46:0/3488768923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ba0105d60 msgr2=0x7f7ba01013e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:12.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.847+0000 7f7ba72d3640 1 --2- 172.21.15.46:0/3488768923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ba0105d60 0x7f7ba01013e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7b90031500 tx=0x7f7b90003d50 comp rx=0 tx=0).stop 2024-04-26T21:34:12.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.847+0000 7f7ba72d3640 1 -- 172.21.15.46:0/3488768923 shutdown_connections 2024-04-26T21:34:12.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.847+0000 7f7ba72d3640 1 --2- 172.21.15.46:0/3488768923 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7b8c0a3c40 0x7f7b8c0a6100 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.847+0000 7f7ba72d3640 1 --2- 172.21.15.46:0/3488768923 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ba0108690 0x7f7ba0101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.847+0000 7f7ba72d3640 1 --2- 172.21.15.46:0/3488768923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ba0105d60 0x7f7ba01013e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.847+0000 7f7ba72d3640 1 --2- 172.21.15.46:0/3488768923 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7ba0103430 0x7f7ba0102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.847+0000 7f7ba72d3640 1 -- 172.21.15.46:0/3488768923 >> 172.21.15.46:0/3488768923 conn(0x7f7ba00fce20 msgr2=0x7f7ba0103cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:12.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.847+0000 7f7ba72d3640 1 -- 172.21.15.46:0/3488768923 shutdown_connections 2024-04-26T21:34:12.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.847+0000 7f7ba72d3640 1 -- 172.21.15.46:0/3488768923 wait complete. 2024-04-26T21:34:12.860 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:34:12.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:12 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/77273598' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:34:12.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:12 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/1286504339' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:34:12.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:12 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/511565703' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:34:12.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.962+0000 7f8439703640 1 -- 172.21.15.46:0/3186717121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8434103b90 msgr2=0x7f8434104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.962+0000 7f8439703640 1 --2- 172.21.15.46:0/3186717121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8434103b90 0x7f8434104010 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8428009900 tx=0x7f842802a410 comp rx=0 tx=0).stop 2024-04-26T21:34:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.962+0000 7f8439703640 1 -- 172.21.15.46:0/3186717121 shutdown_connections 2024-04-26T21:34:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.962+0000 7f8439703640 1 --2- 172.21.15.46:0/3186717121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8434104550 0x7f843410ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.962+0000 7f8439703640 1 --2- 172.21.15.46:0/3186717121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8434103b90 0x7f8434104010 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.962+0000 7f8439703640 1 --2- 172.21.15.46:0/3186717121 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84341029a0 0x7f8434102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.962+0000 7f8439703640 1 -- 172.21.15.46:0/3186717121 >> 172.21.15.46:0/3186717121 conn(0x7f84340fdf30 msgr2=0x7f8434100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.962+0000 7f8439703640 1 -- 172.21.15.46:0/3186717121 shutdown_connections 2024-04-26T21:34:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.962+0000 7f8439703640 1 -- 172.21.15.46:0/3186717121 wait complete. 2024-04-26T21:34:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.962+0000 7f8439703640 1 Processor -- start 2024-04-26T21:34:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.962+0000 7f8439703640 1 -- start start 2024-04-26T21:34:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.963+0000 7f8439703640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84341029a0 0x7f8434198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.963+0000 7f8439703640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8434103b90 0x7f84341991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:12.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.963+0000 7f8439703640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8434104550 0x7f84341a01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:12.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.963+0000 7f8439703640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f843410e530 con 0x7f8434104550 2024-04-26T21:34:12.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.963+0000 7f8439703640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f843410e3b0 con 0x7f84341029a0 2024-04-26T21:34:12.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.963+0000 7f8439703640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f843410e6b0 con 0x7f8434103b90 2024-04-26T21:34:12.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.963+0000 7f84337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8434104550 0x7f84341a01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:12.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.963+0000 7f84337fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8434104550 0x7f84341a01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40258/0 (socket says 172.21.15.46:40258) 2024-04-26T21:34:12.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.963+0000 7f84337fe640 1 -- 172.21.15.46:0/1483519770 learned_addr learned my addr 172.21.15.46:0/1483519770 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:12.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.963+0000 7f8432ffd640 1 --2- 172.21.15.46:0/1483519770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84341029a0 0x7f8434198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:12.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.963+0000 7f84327fc640 1 --2- 172.21.15.46:0/1483519770 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8434103b90 0x7f84341991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:12.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.963+0000 7f84337fe640 1 -- 172.21.15.46:0/1483519770 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8434103b90 msgr2=0x7f84341991c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:12.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.963+0000 7f84337fe640 1 --2- 172.21.15.46:0/1483519770 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8434103b90 0x7f84341991c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.963+0000 7f84337fe640 1 -- 172.21.15.46:0/1483519770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84341029a0 msgr2=0x7f8434198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:12.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.963+0000 7f84337fe640 1 --2- 172.21.15.46:0/1483519770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84341029a0 0x7f8434198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:12.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.963+0000 7f84337fe640 1 -- 172.21.15.46:0/1483519770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84341a0890 con 0x7f8434104550 2024-04-26T21:34:12.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.963+0000 7f84337fe640 1 --2- 172.21.15.46:0/1483519770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8434104550 0x7f84341a01f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f842c00b970 tx=0x7f842c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:12.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.963+0000 7f841bfff640 1 -- 172.21.15.46:0/1483519770 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f842c019070 con 0x7f8434104550 2024-04-26T21:34:12.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.963+0000 7f841bfff640 1 -- 172.21.15.46:0/1483519770 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f842c014440 con 0x7f8434104550 2024-04-26T21:34:12.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.963+0000 7f8439703640 1 -- 172.21.15.46:0/1483519770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84341a0b80 con 0x7f8434104550 2024-04-26T21:34:12.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.963+0000 7f841bfff640 1 -- 172.21.15.46:0/1483519770 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f842c0075d0 con 0x7f8434104550 2024-04-26T21:34:12.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.963+0000 7f8439703640 1 -- 172.21.15.46:0/1483519770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84341a1080 con 0x7f8434104550 2024-04-26T21:34:12.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.964+0000 7f8439703640 1 -- 172.21.15.46:0/1483519770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8434109d40 con 0x7f8434104550 2024-04-26T21:34:12.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.966+0000 7f841bfff640 1 -- 172.21.15.46:0/1483519770 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f842c01c070 con 0x7f8434104550 2024-04-26T21:34:12.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.966+0000 7f841bfff640 1 --2- 172.21.15.46:0/1483519770 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f841c0a3c40 0x7f841c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:12.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.967+0000 7f8432ffd640 1 --2- 172.21.15.46:0/1483519770 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f841c0a3c40 0x7f841c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:12.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.967+0000 7f841bfff640 1 -- 172.21.15.46:0/1483519770 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f842c0cfa80 con 0x7f8434104550 2024-04-26T21:34:12.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.967+0000 7f8432ffd640 1 --2- 172.21.15.46:0/1483519770 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f841c0a3c40 0x7f841c0a6100 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f843419a150 tx=0x7f842400a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:12.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:12.971+0000 7f841bfff640 1 -- 172.21.15.46:0/1483519770 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f842c09bdf0 con 0x7f8434104550 2024-04-26T21:34:13.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:12 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/77273598' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:34:13.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:12 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/1286504339' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:34:13.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:12 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/511565703' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:34:13.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.162+0000 7f8439703640 1 -- 172.21.15.46:0/1483519770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f843402dcc0 con 0x7f8434104550 2024-04-26T21:34:13.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.164+0000 7f841bfff640 1 -- 172.21.15.46:0/1483519770 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22602 (secure 0 0 0) 0x7f842c0a00f0 con 0x7f8434104550 2024-04-26T21:34:13.165 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:34:13.165 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":87,"fsid":"36c7013a-0413-11ef-bc93-c7b262605968","created":"2024-04-26T21:24:30.481156+0000","modified":"2024-04-26T21:33:31.051554+0000","last_up_change":"2024-04-26T21:31:50.100491+0000","last_in_change":"2024-04-26T21:31:29.643428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:27:57.935988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:32:42.195211+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:32:43.162114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f53c8702-e4da-435f-a9dc-3911d0d36ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6803","nonce":2762665125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6805","nonce":2762665125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6809","nonce":2762665125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6807","nonce":2762665125}]},"public_addr":"172.21.15.46:6803/2762665125","cluster_addr":"172.21.15.46:6805/2762665125","heartbeat_back_addr":"172.21.15.46:6809/2762665125","heartbeat_front_addr":"172.21.15.46:6807/2762665125","state":["exists","up"]},{"osd":1,"uuid":"fc4d6f52-fc8a-4efd-80c9-c092b57e7f42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6801","nonce":1628008970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6803","nonce":1628008970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6807","nonce":1628008970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6805","nonce":1628008970}]},"public_addr":"172.21.15.50:6801/1628008970","cluster_addr":"172.21.15.50:6803/1628008970","heartbeat_back_addr":"172.21.15.50:6807/1628008970","heartbeat_front_addr":"172.21.15.50:6805/1628008970","state":["exists","up"]},{"osd":2,"uuid":"26d1baa1-4ea1-4b7e-9fcf-4f8bbf7329c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6801","nonce":1513279137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6803","nonce":1513279137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6807","nonce":1513279137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6805","nonce":1513279137}]},"public_addr":"172.21.15.142:6801/1513279137","cluster_addr":"172.21.15.142:6803/1513279137","heartbeat_back_addr":"172.21.15.142:6807/1513279137","heartbeat_front_addr":"172.21.15.142:6805/1513279137","state":["exists","up"]},{"osd":3,"uuid":"34d2401f-5fcc-491c-8fcf-c3c492611f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6811","nonce":2435181760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6813","nonce":2435181760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6817","nonce":2435181760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6815","nonce":2435181760}]},"public_addr":"172.21.15.46:6811/2435181760","cluster_addr":"172.21.15.46:6813/2435181760","heartbeat_back_addr":"172.21.15.46:6817/2435181760","heartbeat_front_addr":"172.21.15.46:6815/2435181760","state":["exists","up"]},{"osd":4,"uuid":"986bb709-e4cf-400d-982e-c3f5ac413e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6809","nonce":3177333250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6811","nonce":3177333250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6815","nonce":3177333250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6813","nonce":3177333250}]},"public_addr":"172.21.15.50:6809/3177333250","cluster_addr":"172.21.15.50:6811/3177333250","heartbeat_back_addr":"172.21.15.50:6815/3177333250","heartbeat_front_addr":"172.21.15.50:6813/3177333250","state":["exists","up"]},{"osd":5,"uuid":"b7d65851-ebe1-45e2-833d-8022446f2698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6809","nonce":4113107737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6811","nonce":4113107737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6815","nonce":4113107737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6813","nonce":4113107737}]},"public_addr":"172.21.15.142:6809/4113107737","cluster_addr":"172.21.15.142:6811/4113107737","heartbeat_back_addr":"172.21.15.142:6815/4113107737","heartbeat_front_addr":"172.21.15.142:6813/4113107737","state":["exists","up"]},{"osd":6,"uuid":"2a3e96d7-d5bf-44ff-9134-29f1420c0e4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6819","nonce":2325162882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6821","nonce":2325162882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6825","nonce":2325162882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6823","nonce":2325162882}]},"public_addr":"172.21.15.46:6819/2325162882","cluster_addr":"172.21.15.46:6821/2325162882","heartbeat_back_addr":"172.21.15.46:6825/2325162882","heartbeat_front_addr":"172.21.15.46:6823/2325162882","state":["exists","up"]},{"osd":7,"uuid":"e53ae885-23b6-46fd-b4b1-c62f31a6921f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6817","nonce":820310228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6819","nonce":820310228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6823","nonce":820310228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6821","nonce":820310228}]},"public_addr":"172.21.15.50:6817/820310228","cluster_addr":"172.21.15.50:6819/820310228","heartbeat_back_addr":"172.21.15.50:6823/820310228","heartbeat_front_addr":"172.21.15.50:6821/820310228","state":["exists","up"]},{"osd":8,"uuid":"356e2692-16a9-4045-b58f-c8f9cb5d7241","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6817","nonce":1824918711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6819","nonce":1824918711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6823","nonce":1824918711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6821","nonce":1824918711}]},"public_addr":"172.21.15.142:6817/1824918711","cluster_addr":"172.21.15.142:6819/1824918711","heartbeat_back_addr":"172.21.15.142:6823/1824918711","heartbeat_front_addr":"172.21.15.142:6821/1824918711","state":["exists","up"]},{"osd":9,"uuid":"8e55570e-77e5-4892-8b9f-77dcd03a4eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6827","nonce":3601947736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6829","nonce":3601947736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6833","nonce":3601947736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6831","nonce":3601947736}]},"public_addr":"172.21.15.46:6827/3601947736","cluster_addr":"172.21.15.46:6829/3601947736","heartbeat_back_addr":"172.21.15.46:6833/3601947736","heartbeat_front_addr":"172.21.15.46:6831/3601947736","state":["exists","up"]},{"osd":10,"uuid":"366389fa-fb9a-408b-b5c2-343c054c5a64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6825","nonce":3135765006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6827","nonce":3135765006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6831","nonce":3135765006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6829","nonce":3135765006}]},"public_addr":"172.21.15.50:6825/3135765006","cluster_addr":"172.21.15.50:6827/3135765006","heartbeat_back_addr":"172.21.15.50:6831/3135765006","heartbeat_front_addr":"172.21.15.50:6829/3135765006","state":["exists","up"]},{"osd":11,"uuid":"7b6e3d93-59e1-4853-b4a2-261b728c47a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6825","nonce":1482712612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6827","nonce":1482712612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6831","nonce":1482712612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6829","nonce":1482712612}]},"public_addr":"172.21.15.142:6825/1482712612","cluster_addr":"172.21.15.142:6827/1482712612","heartbeat_back_addr":"172.21.15.142:6831/1482712612","heartbeat_front_addr":"172.21.15.142:6829/1482712612","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:08.082015+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:32.634928+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:55.978864+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:21.796863+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:46.862518+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:11.440432+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:37.764777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:03.374958+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:28.714052+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:55.761167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:22.330951+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:47.839334+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/2465242088":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6801/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6800/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/1381997090":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/3507268217":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2570024356":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/2691061938":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2775954801":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6800/2896098005":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6801/2896098005":"2024-04-27T21:25:02.078730+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:34:13.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.166+0000 7f8439703640 1 -- 172.21.15.46:0/1483519770 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f841c0a3c40 msgr2=0x7f841c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:13.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.166+0000 7f8439703640 1 --2- 172.21.15.46:0/1483519770 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f841c0a3c40 0x7f841c0a6100 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f843419a150 tx=0x7f842400a2d0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.166+0000 7f8439703640 1 -- 172.21.15.46:0/1483519770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8434104550 msgr2=0x7f84341a01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:13.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.166+0000 7f8439703640 1 --2- 172.21.15.46:0/1483519770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8434104550 0x7f84341a01f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f842c00b970 tx=0x7f842c00be30 comp rx=0 tx=0).stop 2024-04-26T21:34:13.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.166+0000 7f8439703640 1 -- 172.21.15.46:0/1483519770 shutdown_connections 2024-04-26T21:34:13.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.166+0000 7f8439703640 1 --2- 172.21.15.46:0/1483519770 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f841c0a3c40 0x7f841c0a6100 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.166+0000 7f8439703640 1 --2- 172.21.15.46:0/1483519770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8434104550 0x7f84341a01f0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.166+0000 7f8439703640 1 --2- 172.21.15.46:0/1483519770 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8434103b90 0x7f84341991c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.166+0000 7f8439703640 1 --2- 172.21.15.46:0/1483519770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f84341029a0 0x7f8434198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.166+0000 7f8439703640 1 -- 172.21.15.46:0/1483519770 >> 172.21.15.46:0/1483519770 conn(0x7f84340fdf30 msgr2=0x7f8434106f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:13.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.166+0000 7f8439703640 1 -- 172.21.15.46:0/1483519770 shutdown_connections 2024-04-26T21:34:13.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.166+0000 7f8439703640 1 -- 172.21.15.46:0/1483519770 wait complete. 2024-04-26T21:34:13.179 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T21:27:57.935988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T21:32:42.195211+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T21:32:43.162114+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T21:34:13.179 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T21:34:13.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.283+0000 7f5675540640 1 -- 172.21.15.46:0/2623329863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56701034b0 msgr2=0x7f56701058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:13.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.283+0000 7f5675540640 1 --2- 172.21.15.46:0/2623329863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56701034b0 0x7f56701058a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5660009ca0 tx=0x7f566002a8c0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.283+0000 7f5675540640 1 -- 172.21.15.46:0/2623329863 shutdown_connections 2024-04-26T21:34:13.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.283+0000 7f5675540640 1 --2- 172.21.15.46:0/2623329863 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5670108710 0x7f567010ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.283+0000 7f5675540640 1 --2- 172.21.15.46:0/2623329863 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5670105de0 0x7f56701081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.283+0000 7f5675540640 1 --2- 172.21.15.46:0/2623329863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56701034b0 0x7f56701058a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.283+0000 7f5675540640 1 -- 172.21.15.46:0/2623329863 >> 172.21.15.46:0/2623329863 conn(0x7f56700fce20 msgr2=0x7f56700ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:13.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.283+0000 7f5675540640 1 -- 172.21.15.46:0/2623329863 shutdown_connections 2024-04-26T21:34:13.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.283+0000 7f5675540640 1 -- 172.21.15.46:0/2623329863 wait complete. 2024-04-26T21:34:13.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.284+0000 7f5675540640 1 Processor -- start 2024-04-26T21:34:13.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.284+0000 7f5675540640 1 -- start start 2024-04-26T21:34:13.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.284+0000 7f5675540640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56701034b0 0x7f5670198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:13.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.284+0000 7f5675540640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5670105de0 0x7f56701991e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:13.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.284+0000 7f5675540640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5670108710 0x7f56701a0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:13.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.284+0000 7f5675540640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f567010e530 con 0x7f56701034b0 2024-04-26T21:34:13.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.284+0000 7f5675540640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f567010e3b0 con 0x7f5670108710 2024-04-26T21:34:13.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.284+0000 7f5675540640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f567010e6b0 con 0x7f5670105de0 2024-04-26T21:34:13.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.284+0000 7f566effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56701034b0 0x7f5670198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:13.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.284+0000 7f566effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56701034b0 0x7f5670198ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40276/0 (socket says 172.21.15.46:40276) 2024-04-26T21:34:13.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.284+0000 7f566effd640 1 -- 172.21.15.46:0/47003149 learned_addr learned my addr 172.21.15.46:0/47003149 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:13.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.284+0000 7f566f7fe640 1 --2- 172.21.15.46:0/47003149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5670108710 0x7f56701a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:13.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.284+0000 7f566effd640 1 -- 172.21.15.46:0/47003149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5670105de0 msgr2=0x7f56701991e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:13.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.284+0000 7f566effd640 1 --2- 172.21.15.46:0/47003149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5670105de0 0x7f56701991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.284+0000 7f566effd640 1 -- 172.21.15.46:0/47003149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5670108710 msgr2=0x7f56701a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:13.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.284+0000 7f566effd640 1 --2- 172.21.15.46:0/47003149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5670108710 0x7f56701a0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.284+0000 7f566effd640 1 -- 172.21.15.46:0/47003149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56701a08e0 con 0x7f56701034b0 2024-04-26T21:34:13.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.285+0000 7f566effd640 1 --2- 172.21.15.46:0/47003149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56701034b0 0x7f5670198ca0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5660006010 tx=0x7f5660003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:13.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.285+0000 7f5657fff640 1 -- 172.21.15.46:0/47003149 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56600317c0 con 0x7f56701034b0 2024-04-26T21:34:13.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.285+0000 7f5675540640 1 -- 172.21.15.46:0/47003149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56701a0b70 con 0x7f56701034b0 2024-04-26T21:34:13.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.285+0000 7f5657fff640 1 -- 172.21.15.46:0/47003149 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56600044f0 con 0x7f56701034b0 2024-04-26T21:34:13.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.285+0000 7f5657fff640 1 -- 172.21.15.46:0/47003149 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f566002fd10 con 0x7f56701034b0 2024-04-26T21:34:13.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.285+0000 7f5675540640 1 -- 172.21.15.46:0/47003149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56701a10c0 con 0x7f56701034b0 2024-04-26T21:34:13.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.286+0000 7f5675540640 1 -- 172.21.15.46:0/47003149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56700fe910 con 0x7f56701034b0 2024-04-26T21:34:13.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.287+0000 7f5657fff640 1 -- 172.21.15.46:0/47003149 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5660004040 con 0x7f56701034b0 2024-04-26T21:34:13.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.287+0000 7f5657fff640 1 --2- 172.21.15.46:0/47003149 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f56580a3c40 0x7f56580a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:13.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.288+0000 7f566e7fc640 1 --2- 172.21.15.46:0/47003149 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f56580a3c40 0x7f56580a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:13.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.288+0000 7f5657fff640 1 -- 172.21.15.46:0/47003149 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f56600f1a90 con 0x7f56701034b0 2024-04-26T21:34:13.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.288+0000 7f566e7fc640 1 --2- 172.21.15.46:0/47003149 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f56580a3c40 0x7f56580a6100 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f567019a1a0 tx=0x7f5664006c90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:13.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.292+0000 7f5657fff640 1 -- 172.21.15.46:0/47003149 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f56600be450 con 0x7f56701034b0 2024-04-26T21:34:13.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.477+0000 7f5675540640 1 -- 172.21.15.46:0/47003149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f567002dd20 con 0x7f56701034b0 2024-04-26T21:34:13.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.478+0000 7f5657fff640 1 -- 172.21.15.46:0/47003149 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f56600bdd50 con 0x7f56701034b0 2024-04-26T21:34:13.479 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-26T21:34:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.480+0000 7f5675540640 1 -- 172.21.15.46:0/47003149 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f56580a3c40 msgr2=0x7f56580a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.480+0000 7f5675540640 1 --2- 172.21.15.46:0/47003149 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f56580a3c40 0x7f56580a6100 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f567019a1a0 tx=0x7f5664006c90 comp rx=0 tx=0).stop 2024-04-26T21:34:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.480+0000 7f5675540640 1 -- 172.21.15.46:0/47003149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56701034b0 msgr2=0x7f5670198ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.480+0000 7f5675540640 1 --2- 172.21.15.46:0/47003149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56701034b0 0x7f5670198ca0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5660006010 tx=0x7f5660003ba0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.480+0000 7f5675540640 1 -- 172.21.15.46:0/47003149 shutdown_connections 2024-04-26T21:34:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.480+0000 7f5675540640 1 --2- 172.21.15.46:0/47003149 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f56580a3c40 0x7f56580a6100 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.480+0000 7f5675540640 1 --2- 172.21.15.46:0/47003149 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5670108710 0x7f56701a0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.480+0000 7f5675540640 1 --2- 172.21.15.46:0/47003149 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5670105de0 0x7f56701991e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.480+0000 7f5675540640 1 --2- 172.21.15.46:0/47003149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56701034b0 0x7f5670198ca0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.480+0000 7f5675540640 1 -- 172.21.15.46:0/47003149 >> 172.21.15.46:0/47003149 conn(0x7f56700fce20 msgr2=0x7f5670103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.480+0000 7f5675540640 1 -- 172.21.15.46:0/47003149 shutdown_connections 2024-04-26T21:34:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.480+0000 7f5675540640 1 -- 172.21.15.46:0/47003149 wait complete. 2024-04-26T21:34:13.492 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T21:34:13.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.596+0000 7fa3a00da640 1 -- 172.21.15.46:0/442117763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa398103440 msgr2=0x7fa398109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:13.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.596+0000 7fa3a00da640 1 --2- 172.21.15.46:0/442117763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa398103440 0x7fa398109cd0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa394009ca0 tx=0x7fa39402a8c0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.596+0000 7fa3a00da640 1 -- 172.21.15.46:0/442117763 shutdown_connections 2024-04-26T21:34:13.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.596+0000 7fa3a00da640 1 --2- 172.21.15.46:0/442117763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa398103440 0x7fa398109cd0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.596+0000 7fa3a00da640 1 --2- 172.21.15.46:0/442117763 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa398102a80 0x7fa398102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.596+0000 7fa3a00da640 1 --2- 172.21.15.46:0/442117763 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa398101890 0x7fa398101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.596+0000 7fa3a00da640 1 -- 172.21.15.46:0/442117763 >> 172.21.15.46:0/442117763 conn(0x7fa3980fce20 msgr2=0x7fa3980ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:13.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.596+0000 7fa3a00da640 1 -- 172.21.15.46:0/442117763 shutdown_connections 2024-04-26T21:34:13.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.596+0000 7fa3a00da640 1 -- 172.21.15.46:0/442117763 wait complete. 2024-04-26T21:34:13.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.597+0000 7fa3a00da640 1 Processor -- start 2024-04-26T21:34:13.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.597+0000 7fa3a00da640 1 -- start start 2024-04-26T21:34:13.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.597+0000 7fa3a00da640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa398101890 0x7fa398198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:13.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.597+0000 7fa3a00da640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa398102a80 0x7fa3981991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:13.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.597+0000 7fa3a00da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa398103440 0x7fa3981a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:13.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.597+0000 7fa3a00da640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa39810d3c0 con 0x7fa398103440 2024-04-26T21:34:13.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.597+0000 7fa3a00da640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa39810d240 con 0x7fa398101890 2024-04-26T21:34:13.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.597+0000 7fa3a00da640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa39810d540 con 0x7fa398102a80 2024-04-26T21:34:13.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.597+0000 7fa39e677640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa398103440 0x7fa3981a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:13.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.597+0000 7fa39e677640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa398103440 0x7fa3981a01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40292/0 (socket says 172.21.15.46:40292) 2024-04-26T21:34:13.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.597+0000 7fa39e677640 1 -- 172.21.15.46:0/2060112083 learned_addr learned my addr 172.21.15.46:0/2060112083 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:13.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.597+0000 7fa39d675640 1 --2- 172.21.15.46:0/2060112083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa398102a80 0x7fa3981991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:13.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.597+0000 7fa39de76640 1 --2- 172.21.15.46:0/2060112083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa398101890 0x7fa398198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:13.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.598+0000 7fa39e677640 1 -- 172.21.15.46:0/2060112083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa398102a80 msgr2=0x7fa3981991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:13.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.598+0000 7fa39e677640 1 --2- 172.21.15.46:0/2060112083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa398102a80 0x7fa3981991b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.598+0000 7fa39e677640 1 -- 172.21.15.46:0/2060112083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa398101890 msgr2=0x7fa398198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:13.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.598+0000 7fa39e677640 1 --2- 172.21.15.46:0/2060112083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa398101890 0x7fa398198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.598+0000 7fa39e677640 1 -- 172.21.15.46:0/2060112083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3981a0880 con 0x7fa398103440 2024-04-26T21:34:13.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.598+0000 7fa39e677640 1 --2- 172.21.15.46:0/2060112083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa398103440 0x7fa3981a01e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa3940097b0 tx=0x7fa394003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:13.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.600+0000 7fa38effd640 1 -- 172.21.15.46:0/2060112083 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa394004240 con 0x7fa398103440 2024-04-26T21:34:13.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.600+0000 7fa38effd640 1 -- 172.21.15.46:0/2060112083 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa39402f380 con 0x7fa398103440 2024-04-26T21:34:13.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.600+0000 7fa38effd640 1 -- 172.21.15.46:0/2060112083 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa39402fc30 con 0x7fa398103440 2024-04-26T21:34:13.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.600+0000 7fa3a00da640 1 -- 172.21.15.46:0/2060112083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3981a0b10 con 0x7fa398103440 2024-04-26T21:34:13.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.600+0000 7fa3a00da640 1 -- 172.21.15.46:0/2060112083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3981a10b0 con 0x7fa398103440 2024-04-26T21:34:13.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.601+0000 7fa3a00da640 1 -- 172.21.15.46:0/2060112083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3981a1280 con 0x7fa398103440 2024-04-26T21:34:13.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.602+0000 7fa38effd640 1 -- 172.21.15.46:0/2060112083 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa3940043e0 con 0x7fa398103440 2024-04-26T21:34:13.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.603+0000 7fa38effd640 1 --2- 172.21.15.46:0/2060112083 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fa3840a3cc0 0x7fa3840a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:13.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.603+0000 7fa39de76640 1 --2- 172.21.15.46:0/2060112083 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fa3840a3cc0 0x7fa3840a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:13.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.603+0000 7fa38effd640 1 -- 172.21.15.46:0/2060112083 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7fa3940f1b50 con 0x7fa398103440 2024-04-26T21:34:13.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.604+0000 7fa39de76640 1 --2- 172.21.15.46:0/2060112083 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fa3840a3cc0 0x7fa3840a6180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa3900058c0 tx=0x7fa39000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:13.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.608+0000 7fa38effd640 1 -- 172.21.15.46:0/2060112083 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa3940be4b0 con 0x7fa398103440 2024-04-26T21:34:13.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.794+0000 7fa3a00da640 1 -- 172.21.15.46:0/2060112083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa3981a1450 con 0x7fa398103440 2024-04-26T21:34:13.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.795+0000 7fa38effd640 1 -- 172.21.15.46:0/2060112083 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7fa3940098b0 con 0x7fa398103440 2024-04-26T21:34:13.796 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-26T21:34:13.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.797+0000 7fa3a00da640 1 -- 172.21.15.46:0/2060112083 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fa3840a3cc0 msgr2=0x7fa3840a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:13.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.797+0000 7fa3a00da640 1 --2- 172.21.15.46:0/2060112083 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fa3840a3cc0 0x7fa3840a6180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa3900058c0 tx=0x7fa39000a2d0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.797+0000 7fa3a00da640 1 -- 172.21.15.46:0/2060112083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa398103440 msgr2=0x7fa3981a01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:13.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.797+0000 7fa3a00da640 1 --2- 172.21.15.46:0/2060112083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa398103440 0x7fa3981a01e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa3940097b0 tx=0x7fa394003c20 comp rx=0 tx=0).stop 2024-04-26T21:34:13.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.797+0000 7fa3a00da640 1 -- 172.21.15.46:0/2060112083 shutdown_connections 2024-04-26T21:34:13.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.797+0000 7fa3a00da640 1 --2- 172.21.15.46:0/2060112083 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fa3840a3cc0 0x7fa3840a6180 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.797+0000 7fa3a00da640 1 --2- 172.21.15.46:0/2060112083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa398103440 0x7fa3981a01e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.797+0000 7fa3a00da640 1 --2- 172.21.15.46:0/2060112083 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa398102a80 0x7fa3981991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.797+0000 7fa3a00da640 1 --2- 172.21.15.46:0/2060112083 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa398101890 0x7fa398198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.797+0000 7fa3a00da640 1 -- 172.21.15.46:0/2060112083 >> 172.21.15.46:0/2060112083 conn(0x7fa3980fce20 msgr2=0x7fa3980fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:13.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.797+0000 7fa3a00da640 1 -- 172.21.15.46:0/2060112083 shutdown_connections 2024-04-26T21:34:13.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.797+0000 7fa3a00da640 1 -- 172.21.15.46:0/2060112083 wait complete. 2024-04-26T21:34:13.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:13 smithi046 ceph-mon[29201]: pgmap v338: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:34:13.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:13 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3488768923' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:34:13.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:13 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1483519770' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:34:13.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:13 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/47003149' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:34:13.810 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T21:34:13.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:13 smithi050 ceph-mon[32753]: pgmap v338: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:34:13.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:13 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3488768923' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:34:13.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:13 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/1483519770' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:34:13.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:13 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/47003149' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:34:13.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.914+0000 7f28fca52640 1 -- 172.21.15.46:0/3103642935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f8103b80 msgr2=0x7f28f8104000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:13.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.914+0000 7f28fca52640 1 --2- 172.21.15.46:0/3103642935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f8103b80 0x7f28f8104000 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f28e0009900 tx=0x7f28e002a410 comp rx=0 tx=0).stop 2024-04-26T21:34:13.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.914+0000 7f28fca52640 1 -- 172.21.15.46:0/3103642935 shutdown_connections 2024-04-26T21:34:13.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.914+0000 7f28fca52640 1 --2- 172.21.15.46:0/3103642935 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28f8104540 0x7f28f810add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.914+0000 7f28fca52640 1 --2- 172.21.15.46:0/3103642935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f8103b80 0x7f28f8104000 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.914+0000 7f28fca52640 1 --2- 172.21.15.46:0/3103642935 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28f8102990 0x7f28f8102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.914+0000 7f28fca52640 1 -- 172.21.15.46:0/3103642935 >> 172.21.15.46:0/3103642935 conn(0x7f28f80fdf00 msgr2=0x7f28f8100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:13.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.914+0000 7f28fca52640 1 -- 172.21.15.46:0/3103642935 shutdown_connections 2024-04-26T21:34:13.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.914+0000 7f28fca52640 1 -- 172.21.15.46:0/3103642935 wait complete. 2024-04-26T21:34:13.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.914+0000 7f28fca52640 1 Processor -- start 2024-04-26T21:34:13.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.915+0000 7f28fca52640 1 -- start start 2024-04-26T21:34:13.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.915+0000 7f28fca52640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f8102990 0x7f28f8198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:13.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.915+0000 7f28fca52640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28f8103b80 0x7f28f8199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:13.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.915+0000 7f28fca52640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28f8104540 0x7f28f81a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:13.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.915+0000 7f28fca52640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f28f810e520 con 0x7f28f8102990 2024-04-26T21:34:13.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.915+0000 7f28fca52640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f28f810e3a0 con 0x7f28f8103b80 2024-04-26T21:34:13.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.915+0000 7f28fca52640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f28f810e6a0 con 0x7f28f8104540 2024-04-26T21:34:13.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.915+0000 7f28f659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f8102990 0x7f28f8198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:13.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.915+0000 7f28f659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f8102990 0x7f28f8198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40304/0 (socket says 172.21.15.46:40304) 2024-04-26T21:34:13.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.915+0000 7f28f659c640 1 -- 172.21.15.46:0/3317719938 learned_addr learned my addr 172.21.15.46:0/3317719938 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:13.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.915+0000 7f28f659c640 1 -- 172.21.15.46:0/3317719938 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28f8104540 msgr2=0x7f28f81a01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:13.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.915+0000 7f28f5d9b640 1 --2- 172.21.15.46:0/3317719938 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28f8103b80 0x7f28f8199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:13.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.915+0000 7f28f659c640 1 --2- 172.21.15.46:0/3317719938 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28f8104540 0x7f28f81a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.915+0000 7f28f659c640 1 -- 172.21.15.46:0/3317719938 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28f8103b80 msgr2=0x7f28f8199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:13.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.915+0000 7f28f659c640 1 --2- 172.21.15.46:0/3317719938 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28f8103b80 0x7f28f8199180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:13.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.915+0000 7f28f659c640 1 -- 172.21.15.46:0/3317719938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28f81a0850 con 0x7f28f8102990 2024-04-26T21:34:13.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.915+0000 7f28f659c640 1 --2- 172.21.15.46:0/3317719938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f8102990 0x7f28f8198c40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f28f000c970 tx=0x7f28f000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:13.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.915+0000 7f28ef7fe640 1 -- 172.21.15.46:0/3317719938 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f0004220 con 0x7f28f8102990 2024-04-26T21:34:13.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.915+0000 7f28ef7fe640 1 -- 172.21.15.46:0/3317719938 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28f00043c0 con 0x7f28f8102990 2024-04-26T21:34:13.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.915+0000 7f28fca52640 1 -- 172.21.15.46:0/3317719938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28f81a0b40 con 0x7f28f8102990 2024-04-26T21:34:13.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.915+0000 7f28ef7fe640 1 -- 172.21.15.46:0/3317719938 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f000f450 con 0x7f28f8102990 2024-04-26T21:34:13.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.916+0000 7f28fca52640 1 -- 172.21.15.46:0/3317719938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28f81a1090 con 0x7f28f8102990 2024-04-26T21:34:13.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.916+0000 7f28fca52640 1 -- 172.21.15.46:0/3317719938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28f8109d50 con 0x7f28f8102990 2024-04-26T21:34:13.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.918+0000 7f28ef7fe640 1 -- 172.21.15.46:0/3317719938 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f28f0003610 con 0x7f28f8102990 2024-04-26T21:34:13.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.918+0000 7f28ef7fe640 1 --2- 172.21.15.46:0/3317719938 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f28e40a3c40 0x7f28e40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:13.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.919+0000 7f28f5d9b640 1 --2- 172.21.15.46:0/3317719938 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f28e40a3c40 0x7f28e40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:13.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.919+0000 7f28ef7fe640 1 -- 172.21.15.46:0/3317719938 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f28f00d0520 con 0x7f28f8102990 2024-04-26T21:34:13.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.919+0000 7f28f5d9b640 1 --2- 172.21.15.46:0/3317719938 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f28e40a3c40 0x7f28e40a6100 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f28e0005c40 tx=0x7f28e0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:13.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:13.923+0000 7f28ef7fe640 1 -- 172.21.15.46:0/3317719938 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f28f0014030 con 0x7f28f8102990 2024-04-26T21:34:13.942 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:13 smithi142 ceph-mon[32944]: pgmap v338: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:34:13.942 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:13 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3488768923' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:34:13.942 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:13 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/1483519770' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:34:13.942 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:13 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/47003149' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:34:14.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.110+0000 7f28fca52640 1 -- 172.21.15.46:0/3317719938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f28f8063590 con 0x7f28f8102990 2024-04-26T21:34:14.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.111+0000 7f28ef7fe640 1 -- 172.21.15.46:0/3317719938 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f28f0098030 con 0x7f28f8102990 2024-04-26T21:34:14.112 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-26T21:34:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.113+0000 7f28fca52640 1 -- 172.21.15.46:0/3317719938 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f28e40a3c40 msgr2=0x7f28e40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.113+0000 7f28fca52640 1 --2- 172.21.15.46:0/3317719938 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f28e40a3c40 0x7f28e40a6100 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f28e0005c40 tx=0x7f28e0000bc0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.113+0000 7f28fca52640 1 -- 172.21.15.46:0/3317719938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f8102990 msgr2=0x7f28f8198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.113+0000 7f28fca52640 1 --2- 172.21.15.46:0/3317719938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f8102990 0x7f28f8198c40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f28f000c970 tx=0x7f28f000ce30 comp rx=0 tx=0).stop 2024-04-26T21:34:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.113+0000 7f28fca52640 1 -- 172.21.15.46:0/3317719938 shutdown_connections 2024-04-26T21:34:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.113+0000 7f28fca52640 1 --2- 172.21.15.46:0/3317719938 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f28e40a3c40 0x7f28e40a6100 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.113+0000 7f28fca52640 1 --2- 172.21.15.46:0/3317719938 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f28f8104540 0x7f28f81a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.113+0000 7f28fca52640 1 --2- 172.21.15.46:0/3317719938 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28f8103b80 0x7f28f8199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.113+0000 7f28fca52640 1 --2- 172.21.15.46:0/3317719938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28f8102990 0x7f28f8198c40 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.113+0000 7f28fca52640 1 -- 172.21.15.46:0/3317719938 >> 172.21.15.46:0/3317719938 conn(0x7f28f80fdf00 msgr2=0x7f28f8106f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.113+0000 7f28fca52640 1 -- 172.21.15.46:0/3317719938 shutdown_connections 2024-04-26T21:34:14.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.113+0000 7f28fca52640 1 -- 172.21.15.46:0/3317719938 wait complete. 2024-04-26T21:34:14.125 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:34:14.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.226+0000 7f04a36b0640 1 -- 172.21.15.46:0/34078961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f049c1029a0 msgr2=0x7f049c102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:14.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.226+0000 7f04a36b0640 1 --2- 172.21.15.46:0/34078961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f049c1029a0 0x7f049c102dc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0494009900 tx=0x7f049402a410 comp rx=0 tx=0).stop 2024-04-26T21:34:14.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.226+0000 7f04a36b0640 1 -- 172.21.15.46:0/34078961 shutdown_connections 2024-04-26T21:34:14.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.226+0000 7f04a36b0640 1 --2- 172.21.15.46:0/34078961 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f049c104550 0x7f049c10ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.226+0000 7f04a36b0640 1 --2- 172.21.15.46:0/34078961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f049c103b90 0x7f049c104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.226+0000 7f04a36b0640 1 --2- 172.21.15.46:0/34078961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f049c1029a0 0x7f049c102dc0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.226+0000 7f04a36b0640 1 -- 172.21.15.46:0/34078961 >> 172.21.15.46:0/34078961 conn(0x7f049c0fdf30 msgr2=0x7f049c100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:14.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.226+0000 7f04a36b0640 1 -- 172.21.15.46:0/34078961 shutdown_connections 2024-04-26T21:34:14.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.226+0000 7f04a36b0640 1 -- 172.21.15.46:0/34078961 wait complete. 2024-04-26T21:34:14.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.226+0000 7f04a36b0640 1 Processor -- start 2024-04-26T21:34:14.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.226+0000 7f04a36b0640 1 -- start start 2024-04-26T21:34:14.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.226+0000 7f04a36b0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f049c1029a0 0x7f049c198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:14.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.227+0000 7f04a36b0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f049c103b90 0x7f049c199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:14.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.227+0000 7f04a36b0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f049c104550 0x7f049c1a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:14.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.227+0000 7f04a36b0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f049c10e4d0 con 0x7f049c103b90 2024-04-26T21:34:14.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.227+0000 7f04a36b0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f049c10e350 con 0x7f049c104550 2024-04-26T21:34:14.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.227+0000 7f04a36b0640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f049c10e650 con 0x7f049c1029a0 2024-04-26T21:34:14.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.227+0000 7f04a0c4b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f049c103b90 0x7f049c199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:14.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.227+0000 7f04a0c4b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f049c103b90 0x7f049c199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40318/0 (socket says 172.21.15.46:40318) 2024-04-26T21:34:14.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.227+0000 7f04a0c4b640 1 -- 172.21.15.46:0/3282322166 learned_addr learned my addr 172.21.15.46:0/3282322166 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:14.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.227+0000 7f04a144c640 1 --2- 172.21.15.46:0/3282322166 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f049c1029a0 0x7f049c198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:14.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.227+0000 7f04a1c4d640 1 --2- 172.21.15.46:0/3282322166 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f049c104550 0x7f049c1a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:14.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.227+0000 7f04a0c4b640 1 -- 172.21.15.46:0/3282322166 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f049c1029a0 msgr2=0x7f049c198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:14.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.227+0000 7f04a0c4b640 1 --2- 172.21.15.46:0/3282322166 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f049c1029a0 0x7f049c198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.227+0000 7f04a0c4b640 1 -- 172.21.15.46:0/3282322166 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f049c104550 msgr2=0x7f049c1a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:14.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.227+0000 7f04a0c4b640 1 --2- 172.21.15.46:0/3282322166 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f049c104550 0x7f049c1a01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.227+0000 7f04a0c4b640 1 -- 172.21.15.46:0/3282322166 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f049c1a0810 con 0x7f049c103b90 2024-04-26T21:34:14.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.227+0000 7f04a0c4b640 1 --2- 172.21.15.46:0/3282322166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f049c103b90 0x7f049c199130 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f048c00d950 tx=0x7f048c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:14.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.227+0000 7f04927fc640 1 -- 172.21.15.46:0/3282322166 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048c017030 con 0x7f049c103b90 2024-04-26T21:34:14.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.227+0000 7f04a36b0640 1 -- 172.21.15.46:0/3282322166 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f049c1a0b00 con 0x7f049c103b90 2024-04-26T21:34:14.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.227+0000 7f04927fc640 1 -- 172.21.15.46:0/3282322166 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f048c004020 con 0x7f049c103b90 2024-04-26T21:34:14.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.227+0000 7f04927fc640 1 -- 172.21.15.46:0/3282322166 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048c0047a0 con 0x7f049c103b90 2024-04-26T21:34:14.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.227+0000 7f04a36b0640 1 -- 172.21.15.46:0/3282322166 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f049c1a10a0 con 0x7f049c103b90 2024-04-26T21:34:14.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.228+0000 7f04a36b0640 1 -- 172.21.15.46:0/3282322166 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f049c109d90 con 0x7f049c103b90 2024-04-26T21:34:14.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.229+0000 7f04927fc640 1 -- 172.21.15.46:0/3282322166 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f048c009a30 con 0x7f049c103b90 2024-04-26T21:34:14.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.230+0000 7f04927fc640 1 --2- 172.21.15.46:0/3282322166 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f04880a3c40 0x7f04880a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:14.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.230+0000 7f04a144c640 1 --2- 172.21.15.46:0/3282322166 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f04880a3c40 0x7f04880a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:14.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.230+0000 7f04927fc640 1 -- 172.21.15.46:0/3282322166 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f048c0d07b0 con 0x7f049c103b90 2024-04-26T21:34:14.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.230+0000 7f04a144c640 1 --2- 172.21.15.46:0/3282322166 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f04880a3c40 0x7f04880a6100 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0494000c00 tx=0x7f04940314a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:14.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.234+0000 7f04927fc640 1 -- 172.21.15.46:0/3282322166 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f048c09d000 con 0x7f049c103b90 2024-04-26T21:34:14.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.465+0000 7f04a36b0640 1 -- 172.21.15.46:0/3282322166 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f049c02dc80 con 0x7f049c103b90 2024-04-26T21:34:14.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.466+0000 7f04927fc640 1 -- 172.21.15.46:0/3282322166 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11863 (secure 0 0 0) 0x7f048c09c900 con 0x7f049c103b90 2024-04-26T21:34:14.467 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:34:14.467 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3674296286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3674296286},{"type":"v1","addr":"172.21.15.46:6835","nonce":3674296286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/391791206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":391791206},{"type":"v1","addr":"172.21.15.46:6837","nonce":391791206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/1222489912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1222489912},{"type":"v1","addr":"172.21.15.46:6839","nonce":1222489912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14658,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/860605548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":860605548},{"type":"v1","addr":"172.21.15.46:6841","nonce":860605548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3621399933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3621399933},{"type":"v1","addr":"172.21.15.142:6833","nonce":3621399933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/345015778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":345015778},{"type":"v1","addr":"172.21.15.50:6833","nonce":345015778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T21:32:44.201541+0000","modified":"2024-04-26T21:32:49.889029+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24526,"mds_2":24509},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/2546509863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":2546509863},{"type":"v1","addr":"172.21.15.50:6835","nonce":2546509863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/3791468066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":3791468066},{"type":"v1","addr":"172.21.15.142:6835","nonce":3791468066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6837/1723132553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1723132553},{"type":"v1","addr":"172.21.15.142:6837","nonce":1723132553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6837/3189620395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3189620395},{"type":"v1","addr":"172.21.15.50:6837","nonce":3189620395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24526":{"gid":24526,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.142:6839/3089509101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3089509101},{"type":"v1","addr":"172.21.15.142:6839","nonce":3089509101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6839/2453253368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2453253368},{"type":"v1","addr":"172.21.15.50:6839","nonce":2453253368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24509,24526,24539]},"id":1}]} 2024-04-26T21:34:14.467 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 22 2024-04-26T21:34:14.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.468+0000 7f04a36b0640 1 -- 172.21.15.46:0/3282322166 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f04880a3c40 msgr2=0x7f04880a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:14.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.468+0000 7f04a36b0640 1 --2- 172.21.15.46:0/3282322166 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f04880a3c40 0x7f04880a6100 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0494000c00 tx=0x7f04940314a0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.468+0000 7f04a36b0640 1 -- 172.21.15.46:0/3282322166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f049c103b90 msgr2=0x7f049c199130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:14.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.468+0000 7f04a36b0640 1 --2- 172.21.15.46:0/3282322166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f049c103b90 0x7f049c199130 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f048c00d950 tx=0x7f048c00de10 comp rx=0 tx=0).stop 2024-04-26T21:34:14.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.468+0000 7f04a36b0640 1 -- 172.21.15.46:0/3282322166 shutdown_connections 2024-04-26T21:34:14.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.468+0000 7f04a36b0640 1 --2- 172.21.15.46:0/3282322166 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f04880a3c40 0x7f04880a6100 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.468+0000 7f04a36b0640 1 --2- 172.21.15.46:0/3282322166 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f049c104550 0x7f049c1a01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.468+0000 7f04a36b0640 1 --2- 172.21.15.46:0/3282322166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f049c103b90 0x7f049c199130 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.468+0000 7f04a36b0640 1 --2- 172.21.15.46:0/3282322166 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f049c1029a0 0x7f049c198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.468+0000 7f04a36b0640 1 -- 172.21.15.46:0/3282322166 >> 172.21.15.46:0/3282322166 conn(0x7f049c0fdf30 msgr2=0x7f049c106f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:14.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.468+0000 7f04a36b0640 1 -- 172.21.15.46:0/3282322166 shutdown_connections 2024-04-26T21:34:14.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.468+0000 7f04a36b0640 1 -- 172.21.15.46:0/3282322166 wait complete. 2024-04-26T21:34:14.481 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:34:14.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.581+0000 7f9253246640 1 -- 172.21.15.46:0/2011137593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f924c102a60 msgr2=0x7f924c102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:14.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.581+0000 7f9253246640 1 --2- 172.21.15.46:0/2011137593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f924c102a60 0x7f924c102ee0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9238009900 tx=0x7f923802a410 comp rx=0 tx=0).stop 2024-04-26T21:34:14.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.581+0000 7f9253246640 1 -- 172.21.15.46:0/2011137593 shutdown_connections 2024-04-26T21:34:14.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.581+0000 7f9253246640 1 --2- 172.21.15.46:0/2011137593 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f924c103420 0x7f924c109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.581+0000 7f9253246640 1 --2- 172.21.15.46:0/2011137593 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f924c102a60 0x7f924c102ee0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.581+0000 7f9253246640 1 --2- 172.21.15.46:0/2011137593 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f924c101870 0x7f924c101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.581+0000 7f9253246640 1 -- 172.21.15.46:0/2011137593 >> 172.21.15.46:0/2011137593 conn(0x7f924c0fcde0 msgr2=0x7f924c0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:14.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.581+0000 7f9253246640 1 -- 172.21.15.46:0/2011137593 shutdown_connections 2024-04-26T21:34:14.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.581+0000 7f9253246640 1 -- 172.21.15.46:0/2011137593 wait complete. 2024-04-26T21:34:14.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.581+0000 7f9253246640 1 Processor -- start 2024-04-26T21:34:14.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.582+0000 7f9253246640 1 -- start start 2024-04-26T21:34:14.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.582+0000 7f9253246640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f924c101870 0x7f924c198d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:14.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.582+0000 7f9253246640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f924c102a60 0x7f924c1992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:14.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.582+0000 7f9253246640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f924c103420 0x7f924c1a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:14.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.582+0000 7f9253246640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f924c10e5a0 con 0x7f924c101870 2024-04-26T21:34:14.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.582+0000 7f9253246640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f924c10e420 con 0x7f924c103420 2024-04-26T21:34:14.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.582+0000 7f9253246640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f924c10e720 con 0x7f924c102a60 2024-04-26T21:34:14.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.582+0000 7f9250fe2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f924c101870 0x7f924c198d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:14.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.582+0000 7f9250fe2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f924c101870 0x7f924c198d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40338/0 (socket says 172.21.15.46:40338) 2024-04-26T21:34:14.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.582+0000 7f9250fe2640 1 -- 172.21.15.46:0/770590354 learned_addr learned my addr 172.21.15.46:0/770590354 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:14.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.582+0000 7f924bfff640 1 --2- 172.21.15.46:0/770590354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f924c102a60 0x7f924c1992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:14.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.582+0000 7f9250fe2640 1 -- 172.21.15.46:0/770590354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f924c102a60 msgr2=0x7f924c1992a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:14.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.582+0000 7f9250fe2640 1 --2- 172.21.15.46:0/770590354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f924c102a60 0x7f924c1992a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.582+0000 7f92517e3640 1 --2- 172.21.15.46:0/770590354 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f924c103420 0x7f924c1a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:14.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.582+0000 7f9250fe2640 1 -- 172.21.15.46:0/770590354 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f924c103420 msgr2=0x7f924c1a0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:14.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.582+0000 7f9250fe2640 1 --2- 172.21.15.46:0/770590354 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f924c103420 0x7f924c1a0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.582+0000 7f9250fe2640 1 -- 172.21.15.46:0/770590354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f924c1a0980 con 0x7f924c101870 2024-04-26T21:34:14.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.582+0000 7f924bfff640 1 --2- 172.21.15.46:0/770590354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f924c102a60 0x7f924c1992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:34:14.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.582+0000 7f92517e3640 1 --2- 172.21.15.46:0/770590354 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f924c103420 0x7f924c1a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:34:14.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.582+0000 7f9250fe2640 1 --2- 172.21.15.46:0/770590354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f924c101870 0x7f924c198d60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f924000b450 tx=0x7f924000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:14.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.582+0000 7f9249ffb640 1 -- 172.21.15.46:0/770590354 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9240015020 con 0x7f924c101870 2024-04-26T21:34:14.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.582+0000 7f9249ffb640 1 -- 172.21.15.46:0/770590354 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9240011d40 con 0x7f924c101870 2024-04-26T21:34:14.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.582+0000 7f9253246640 1 -- 172.21.15.46:0/770590354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f924c1a0c70 con 0x7f924c101870 2024-04-26T21:34:14.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.583+0000 7f9249ffb640 1 -- 172.21.15.46:0/770590354 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924000f340 con 0x7f924c101870 2024-04-26T21:34:14.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.583+0000 7f9253246640 1 -- 172.21.15.46:0/770590354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f924c1a11c0 con 0x7f924c101870 2024-04-26T21:34:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.584+0000 7f9253246640 1 -- 172.21.15.46:0/770590354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f924c108c50 con 0x7f924c101870 2024-04-26T21:34:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.585+0000 7f9249ffb640 1 -- 172.21.15.46:0/770590354 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9240018020 con 0x7f924c101870 2024-04-26T21:34:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.585+0000 7f9249ffb640 1 --2- 172.21.15.46:0/770590354 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f923c0a3cc0 0x7f923c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.585+0000 7f9249ffb640 1 -- 172.21.15.46:0/770590354 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f92400cff10 con 0x7f924c101870 2024-04-26T21:34:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.585+0000 7f924bfff640 1 --2- 172.21.15.46:0/770590354 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f923c0a3cc0 0x7f923c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.586+0000 7f924bfff640 1 --2- 172.21.15.46:0/770590354 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f923c0a3cc0 0x7f923c0a6180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9238006010 tx=0x7f9238000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:14.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.590+0000 7f9249ffb640 1 -- 172.21.15.46:0/770590354 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f924001f020 con 0x7f924c101870 2024-04-26T21:34:14.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.776+0000 7f9253246640 1 -- 172.21.15.46:0/770590354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f924c02dc50 con 0x7f924c101870 2024-04-26T21:34:14.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.778+0000 7f9249ffb640 1 -- 172.21.15.46:0/770590354 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22602 (secure 0 0 0) 0x7f924000d040 con 0x7f924c101870 2024-04-26T21:34:14.779 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:34:14.779 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":87,"fsid":"36c7013a-0413-11ef-bc93-c7b262605968","created":"2024-04-26T21:24:30.481156+0000","modified":"2024-04-26T21:33:31.051554+0000","last_up_change":"2024-04-26T21:31:50.100491+0000","last_in_change":"2024-04-26T21:31:29.643428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:27:57.935988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:32:42.195211+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:32:43.162114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f53c8702-e4da-435f-a9dc-3911d0d36ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6803","nonce":2762665125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6805","nonce":2762665125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6809","nonce":2762665125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6807","nonce":2762665125}]},"public_addr":"172.21.15.46:6803/2762665125","cluster_addr":"172.21.15.46:6805/2762665125","heartbeat_back_addr":"172.21.15.46:6809/2762665125","heartbeat_front_addr":"172.21.15.46:6807/2762665125","state":["exists","up"]},{"osd":1,"uuid":"fc4d6f52-fc8a-4efd-80c9-c092b57e7f42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6801","nonce":1628008970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6803","nonce":1628008970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6807","nonce":1628008970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6805","nonce":1628008970}]},"public_addr":"172.21.15.50:6801/1628008970","cluster_addr":"172.21.15.50:6803/1628008970","heartbeat_back_addr":"172.21.15.50:6807/1628008970","heartbeat_front_addr":"172.21.15.50:6805/1628008970","state":["exists","up"]},{"osd":2,"uuid":"26d1baa1-4ea1-4b7e-9fcf-4f8bbf7329c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6801","nonce":1513279137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6803","nonce":1513279137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6807","nonce":1513279137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6805","nonce":1513279137}]},"public_addr":"172.21.15.142:6801/1513279137","cluster_addr":"172.21.15.142:6803/1513279137","heartbeat_back_addr":"172.21.15.142:6807/1513279137","heartbeat_front_addr":"172.21.15.142:6805/1513279137","state":["exists","up"]},{"osd":3,"uuid":"34d2401f-5fcc-491c-8fcf-c3c492611f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6811","nonce":2435181760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6813","nonce":2435181760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6817","nonce":2435181760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6815","nonce":2435181760}]},"public_addr":"172.21.15.46:6811/2435181760","cluster_addr":"172.21.15.46:6813/2435181760","heartbeat_back_addr":"172.21.15.46:6817/2435181760","heartbeat_front_addr":"172.21.15.46:6815/2435181760","state":["exists","up"]},{"osd":4,"uuid":"986bb709-e4cf-400d-982e-c3f5ac413e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6809","nonce":3177333250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6811","nonce":3177333250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6815","nonce":3177333250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6813","nonce":3177333250}]},"public_addr":"172.21.15.50:6809/3177333250","cluster_addr":"172.21.15.50:6811/3177333250","heartbeat_back_addr":"172.21.15.50:6815/3177333250","heartbeat_front_addr":"172.21.15.50:6813/3177333250","state":["exists","up"]},{"osd":5,"uuid":"b7d65851-ebe1-45e2-833d-8022446f2698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6809","nonce":4113107737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6811","nonce":4113107737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6815","nonce":4113107737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6813","nonce":4113107737}]},"public_addr":"172.21.15.142:6809/4113107737","cluster_addr":"172.21.15.142:6811/4113107737","heartbeat_back_addr":"172.21.15.142:6815/4113107737","heartbeat_front_addr":"172.21.15.142:6813/4113107737","state":["exists","up"]},{"osd":6,"uuid":"2a3e96d7-d5bf-44ff-9134-29f1420c0e4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6819","nonce":2325162882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6821","nonce":2325162882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6825","nonce":2325162882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6823","nonce":2325162882}]},"public_addr":"172.21.15.46:6819/2325162882","cluster_addr":"172.21.15.46:6821/2325162882","heartbeat_back_addr":"172.21.15.46:6825/2325162882","heartbeat_front_addr":"172.21.15.46:6823/2325162882","state":["exists","up"]},{"osd":7,"uuid":"e53ae885-23b6-46fd-b4b1-c62f31a6921f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6817","nonce":820310228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6819","nonce":820310228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6823","nonce":820310228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6821","nonce":820310228}]},"public_addr":"172.21.15.50:6817/820310228","cluster_addr":"172.21.15.50:6819/820310228","heartbeat_back_addr":"172.21.15.50:6823/820310228","heartbeat_front_addr":"172.21.15.50:6821/820310228","state":["exists","up"]},{"osd":8,"uuid":"356e2692-16a9-4045-b58f-c8f9cb5d7241","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6817","nonce":1824918711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6819","nonce":1824918711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6823","nonce":1824918711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6821","nonce":1824918711}]},"public_addr":"172.21.15.142:6817/1824918711","cluster_addr":"172.21.15.142:6819/1824918711","heartbeat_back_addr":"172.21.15.142:6823/1824918711","heartbeat_front_addr":"172.21.15.142:6821/1824918711","state":["exists","up"]},{"osd":9,"uuid":"8e55570e-77e5-4892-8b9f-77dcd03a4eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6827","nonce":3601947736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6829","nonce":3601947736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6833","nonce":3601947736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6831","nonce":3601947736}]},"public_addr":"172.21.15.46:6827/3601947736","cluster_addr":"172.21.15.46:6829/3601947736","heartbeat_back_addr":"172.21.15.46:6833/3601947736","heartbeat_front_addr":"172.21.15.46:6831/3601947736","state":["exists","up"]},{"osd":10,"uuid":"366389fa-fb9a-408b-b5c2-343c054c5a64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6825","nonce":3135765006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6827","nonce":3135765006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6831","nonce":3135765006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6829","nonce":3135765006}]},"public_addr":"172.21.15.50:6825/3135765006","cluster_addr":"172.21.15.50:6827/3135765006","heartbeat_back_addr":"172.21.15.50:6831/3135765006","heartbeat_front_addr":"172.21.15.50:6829/3135765006","state":["exists","up"]},{"osd":11,"uuid":"7b6e3d93-59e1-4853-b4a2-261b728c47a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6825","nonce":1482712612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6827","nonce":1482712612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6831","nonce":1482712612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6829","nonce":1482712612}]},"public_addr":"172.21.15.142:6825/1482712612","cluster_addr":"172.21.15.142:6827/1482712612","heartbeat_back_addr":"172.21.15.142:6831/1482712612","heartbeat_front_addr":"172.21.15.142:6829/1482712612","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:08.082015+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:32.634928+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:55.978864+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:21.796863+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:46.862518+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:11.440432+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:37.764777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:03.374958+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:28.714052+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:55.761167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:22.330951+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:47.839334+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/2465242088":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6801/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6800/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/1381997090":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/3507268217":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2570024356":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/2691061938":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2775954801":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6800/2896098005":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6801/2896098005":"2024-04-27T21:25:02.078730+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:34:14.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.780+0000 7f9253246640 1 -- 172.21.15.46:0/770590354 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f923c0a3cc0 msgr2=0x7f923c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:14.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.780+0000 7f9253246640 1 --2- 172.21.15.46:0/770590354 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f923c0a3cc0 0x7f923c0a6180 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9238006010 tx=0x7f9238000bc0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.780+0000 7f9253246640 1 -- 172.21.15.46:0/770590354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f924c101870 msgr2=0x7f924c198d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:14.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.780+0000 7f9253246640 1 --2- 172.21.15.46:0/770590354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f924c101870 0x7f924c198d60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f924000b450 tx=0x7f924000b910 comp rx=0 tx=0).stop 2024-04-26T21:34:14.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.780+0000 7f9253246640 1 -- 172.21.15.46:0/770590354 shutdown_connections 2024-04-26T21:34:14.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.780+0000 7f9253246640 1 --2- 172.21.15.46:0/770590354 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f923c0a3cc0 0x7f923c0a6180 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.780+0000 7f9253246640 1 --2- 172.21.15.46:0/770590354 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f924c103420 0x7f924c1a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.780+0000 7f9253246640 1 --2- 172.21.15.46:0/770590354 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f924c102a60 0x7f924c1992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.780+0000 7f9253246640 1 --2- 172.21.15.46:0/770590354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f924c101870 0x7f924c198d60 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.780+0000 7f9253246640 1 -- 172.21.15.46:0/770590354 >> 172.21.15.46:0/770590354 conn(0x7f924c0fcde0 msgr2=0x7f924c105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:14.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.780+0000 7f9253246640 1 -- 172.21.15.46:0/770590354 shutdown_connections 2024-04-26T21:34:14.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.780+0000 7f9253246640 1 -- 172.21.15.46:0/770590354 wait complete. 2024-04-26T21:34:14.791 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:14 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2060112083' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:34:14.791 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:14 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3317719938' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:34:14.791 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:14 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3282322166' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:34:14.793 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:34:14.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:14 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/2060112083' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:34:14.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:14 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3317719938' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:34:14.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:14 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3282322166' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:34:14.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.893+0000 7f1069e6b640 1 -- 172.21.15.46:0/2969422639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1064105dc0 msgr2=0x7f10641081b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:14.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.893+0000 7f1069e6b640 1 --2- 172.21.15.46:0/2969422639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1064105dc0 0x7f10641081b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1058009900 tx=0x7f105802a410 comp rx=0 tx=0).stop 2024-04-26T21:34:14.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.893+0000 7f1069e6b640 1 -- 172.21.15.46:0/2969422639 shutdown_connections 2024-04-26T21:34:14.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.893+0000 7f1069e6b640 1 --2- 172.21.15.46:0/2969422639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10641086f0 0x7f106410aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.893+0000 7f1069e6b640 1 --2- 172.21.15.46:0/2969422639 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1064105dc0 0x7f10641081b0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.893+0000 7f1069e6b640 1 --2- 172.21.15.46:0/2969422639 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f1064103490 0x7f1064105880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.893+0000 7f1069e6b640 1 -- 172.21.15.46:0/2969422639 >> 172.21.15.46:0/2969422639 conn(0x7f10640fcde0 msgr2=0x7f10640ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:14.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.893+0000 7f1069e6b640 1 -- 172.21.15.46:0/2969422639 shutdown_connections 2024-04-26T21:34:14.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.893+0000 7f1069e6b640 1 -- 172.21.15.46:0/2969422639 wait complete. 2024-04-26T21:34:14.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.893+0000 7f1069e6b640 1 Processor -- start 2024-04-26T21:34:14.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.894+0000 7f1069e6b640 1 -- start start 2024-04-26T21:34:14.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.894+0000 7f1069e6b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1064103490 0x7f1064102e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:14.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.894+0000 7f1069e6b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1064105dc0 0x7f1064101440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:14.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.894+0000 7f1069e6b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10641086f0 0x7f1064101980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:14.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.894+0000 7f1069e6b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f106410e4f0 con 0x7f1064103490 2024-04-26T21:34:14.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.894+0000 7f1069e6b640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f106410e370 con 0x7f1064105dc0 2024-04-26T21:34:14.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.894+0000 7f1069e6b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f106410e670 con 0x7f10641086f0 2024-04-26T21:34:14.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.894+0000 7f10637fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1064103490 0x7f1064102e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:14.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.894+0000 7f10637fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1064103490 0x7f1064102e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40350/0 (socket says 172.21.15.46:40350) 2024-04-26T21:34:14.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.894+0000 7f10637fe640 1 -- 172.21.15.46:0/1257531182 learned_addr learned my addr 172.21.15.46:0/1257531182 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:14.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.894+0000 7f1063fff640 1 --2- 172.21.15.46:0/1257531182 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10641086f0 0x7f1064101980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:14.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.894+0000 7f10637fe640 1 -- 172.21.15.46:0/1257531182 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10641086f0 msgr2=0x7f1064101980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:14.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.894+0000 7f1062ffd640 1 --2- 172.21.15.46:0/1257531182 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1064105dc0 0x7f1064101440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:14.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.894+0000 7f10637fe640 1 --2- 172.21.15.46:0/1257531182 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10641086f0 0x7f1064101980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.894+0000 7f10637fe640 1 -- 172.21.15.46:0/1257531182 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1064105dc0 msgr2=0x7f1064101440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:14.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.894+0000 7f10637fe640 1 --2- 172.21.15.46:0/1257531182 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1064105dc0 0x7f1064101440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:14.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.894+0000 7f10637fe640 1 -- 172.21.15.46:0/1257531182 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1064102210 con 0x7f1064103490 2024-04-26T21:34:14.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.894+0000 7f1063fff640 1 --2- 172.21.15.46:0/1257531182 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10641086f0 0x7f1064101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:34:14.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.894+0000 7f10637fe640 1 --2- 172.21.15.46:0/1257531182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1064103490 0x7f1064102e00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f105400c920 tx=0x7f105400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:14.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.894+0000 7f1060ff9640 1 -- 172.21.15.46:0/1257531182 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1054007c70 con 0x7f1064103490 2024-04-26T21:34:14.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.894+0000 7f1069e6b640 1 -- 172.21.15.46:0/1257531182 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10641a2cb0 con 0x7f1064103490 2024-04-26T21:34:14.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.895+0000 7f1060ff9640 1 -- 172.21.15.46:0/1257531182 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1054007e10 con 0x7f1064103490 2024-04-26T21:34:14.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.895+0000 7f1060ff9640 1 -- 172.21.15.46:0/1257531182 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1054018820 con 0x7f1064103490 2024-04-26T21:34:14.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.895+0000 7f1069e6b640 1 -- 172.21.15.46:0/1257531182 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10641a31a0 con 0x7f1064103490 2024-04-26T21:34:14.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.896+0000 7f1069e6b640 1 -- 172.21.15.46:0/1257531182 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10640fe8d0 con 0x7f1064103490 2024-04-26T21:34:14.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.897+0000 7f1060ff9640 1 -- 172.21.15.46:0/1257531182 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f105400f450 con 0x7f1064103490 2024-04-26T21:34:14.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.897+0000 7f1060ff9640 1 --2- 172.21.15.46:0/1257531182 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f104c0a3c40 0x7f104c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:14.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.897+0000 7f1062ffd640 1 --2- 172.21.15.46:0/1257531182 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f104c0a3c40 0x7f104c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:14.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.898+0000 7f1060ff9640 1 -- 172.21.15.46:0/1257531182 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f10540d07a0 con 0x7f1064103490 2024-04-26T21:34:14.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.898+0000 7f1062ffd640 1 --2- 172.21.15.46:0/1257531182 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f104c0a3c40 0x7f104c0a6100 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1058031500 tx=0x7f1058000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:14.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:14.902+0000 7f1060ff9640 1 -- 172.21.15.46:0/1257531182 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1054014030 con 0x7f1064103490 2024-04-26T21:34:15.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:14 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/2060112083' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:34:15.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:14 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3317719938' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:34:15.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:14 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3282322166' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:34:15.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.128+0000 7f1069e6b640 1 -- 172.21.15.46:0/1257531182 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f106402dd10 con 0x7f1064103490 2024-04-26T21:34:15.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.129+0000 7f1060ff9640 1 -- 172.21.15.46:0/1257531182 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11863 (secure 0 0 0) 0x7f105400de20 con 0x7f1064103490 2024-04-26T21:34:15.130 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:34:15.130 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3674296286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3674296286},{"type":"v1","addr":"172.21.15.46:6835","nonce":3674296286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/391791206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":391791206},{"type":"v1","addr":"172.21.15.46:6837","nonce":391791206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/1222489912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1222489912},{"type":"v1","addr":"172.21.15.46:6839","nonce":1222489912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14658,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/860605548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":860605548},{"type":"v1","addr":"172.21.15.46:6841","nonce":860605548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3621399933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3621399933},{"type":"v1","addr":"172.21.15.142:6833","nonce":3621399933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/345015778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":345015778},{"type":"v1","addr":"172.21.15.50:6833","nonce":345015778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T21:32:44.201541+0000","modified":"2024-04-26T21:32:49.889029+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24526,"mds_2":24509},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/2546509863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":2546509863},{"type":"v1","addr":"172.21.15.50:6835","nonce":2546509863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/3791468066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":3791468066},{"type":"v1","addr":"172.21.15.142:6835","nonce":3791468066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6837/1723132553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1723132553},{"type":"v1","addr":"172.21.15.142:6837","nonce":1723132553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6837/3189620395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3189620395},{"type":"v1","addr":"172.21.15.50:6837","nonce":3189620395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24526":{"gid":24526,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.142:6839/3089509101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3089509101},{"type":"v1","addr":"172.21.15.142:6839","nonce":3089509101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6839/2453253368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2453253368},{"type":"v1","addr":"172.21.15.50:6839","nonce":2453253368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24509,24526,24539]},"id":1}]} 2024-04-26T21:34:15.130 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 22 2024-04-26T21:34:15.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.131+0000 7f1069e6b640 1 -- 172.21.15.46:0/1257531182 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f104c0a3c40 msgr2=0x7f104c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.131+0000 7f1069e6b640 1 --2- 172.21.15.46:0/1257531182 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f104c0a3c40 0x7f104c0a6100 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1058031500 tx=0x7f1058000bc0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.131+0000 7f1069e6b640 1 -- 172.21.15.46:0/1257531182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1064103490 msgr2=0x7f1064102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.131+0000 7f1069e6b640 1 --2- 172.21.15.46:0/1257531182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1064103490 0x7f1064102e00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f105400c920 tx=0x7f105400cde0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.131+0000 7f1069e6b640 1 -- 172.21.15.46:0/1257531182 shutdown_connections 2024-04-26T21:34:15.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.131+0000 7f1069e6b640 1 --2- 172.21.15.46:0/1257531182 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f104c0a3c40 0x7f104c0a6100 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.131+0000 7f1069e6b640 1 --2- 172.21.15.46:0/1257531182 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f10641086f0 0x7f1064101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.131+0000 7f1069e6b640 1 --2- 172.21.15.46:0/1257531182 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1064105dc0 0x7f1064101440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.131+0000 7f1069e6b640 1 --2- 172.21.15.46:0/1257531182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1064103490 0x7f1064102e00 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.131+0000 7f1069e6b640 1 -- 172.21.15.46:0/1257531182 >> 172.21.15.46:0/1257531182 conn(0x7f10640fcde0 msgr2=0x7f1064103dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:15.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.131+0000 7f1069e6b640 1 -- 172.21.15.46:0/1257531182 shutdown_connections 2024-04-26T21:34:15.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.131+0000 7f1069e6b640 1 -- 172.21.15.46:0/1257531182 wait complete. 2024-04-26T21:34:15.144 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T21:32:44.201541+0000', 'modified': '2024-04-26T21:32:49.889029+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24539, 'mds_1': 24526, 'mds_2': 24509}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24470': {'gid': 24470, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.50:6835/2546509863', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6834', 'nonce': 2546509863}, {'type': 'v1', 'addr': '172.21.15.50:6835', 'nonce': 2546509863}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24493': {'gid': 24493, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.142:6835/3791468066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6834', 'nonce': 3791468066}, {'type': 'v1', 'addr': '172.21.15.142:6835', 'nonce': 3791468066}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24505': {'gid': 24505, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.142:6837/1723132553', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6836', 'nonce': 1723132553}, {'type': 'v1', 'addr': '172.21.15.142:6837', 'nonce': 1723132553}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24509': {'gid': 24509, 'name': 'h', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.50:6837/3189620395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6836', 'nonce': 3189620395}, {'type': 'v1', 'addr': '172.21.15.50:6837', 'nonce': 3189620395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24526': {'gid': 24526, 'name': 'i', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.142:6839/3089509101', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6838', 'nonce': 3089509101}, {'type': 'v1', 'addr': '172.21.15.142:6839', 'nonce': 3089509101}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24539': {'gid': 24539, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.50:6839/2453253368', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6838', 'nonce': 2453253368}, {'type': 'v1', 'addr': '172.21.15.50:6839', 'nonce': 2453253368}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24539, 'qdb_cluster': [24509, 24526, 24539]} 2024-04-26T21:34:15.144 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T21:34:15.144 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-26T21:34:15.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.245+0000 7f7d2dc07640 1 -- 172.21.15.46:0/3796842129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d28103410 msgr2=0x7f7d28105800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.245+0000 7f7d2dc07640 1 --2- 172.21.15.46:0/3796842129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d28103410 0x7f7d28105800 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7d18009ca0 tx=0x7f7d1802a8c0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.245+0000 7f7d2dc07640 1 -- 172.21.15.46:0/3796842129 shutdown_connections 2024-04-26T21:34:15.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.245+0000 7f7d2dc07640 1 --2- 172.21.15.46:0/3796842129 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d28108670 0x7f7d2810aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.245+0000 7f7d2dc07640 1 --2- 172.21.15.46:0/3796842129 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d28105d40 0x7f7d28108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.245+0000 7f7d2dc07640 1 --2- 172.21.15.46:0/3796842129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d28103410 0x7f7d28105800 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.245+0000 7f7d2dc07640 1 -- 172.21.15.46:0/3796842129 >> 172.21.15.46:0/3796842129 conn(0x7f7d280fce20 msgr2=0x7f7d280ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:15.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.245+0000 7f7d2dc07640 1 -- 172.21.15.46:0/3796842129 shutdown_connections 2024-04-26T21:34:15.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.245+0000 7f7d2dc07640 1 -- 172.21.15.46:0/3796842129 wait complete. 2024-04-26T21:34:15.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.246+0000 7f7d2dc07640 1 Processor -- start 2024-04-26T21:34:15.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.246+0000 7f7d2dc07640 1 -- start start 2024-04-26T21:34:15.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.246+0000 7f7d2dc07640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d28103410 0x7f7d28198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.246+0000 7f7d2dc07640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d28105d40 0x7f7d28199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.246+0000 7f7d2dc07640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d28108670 0x7f7d281a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.246+0000 7f7d2dc07640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7d2810e510 con 0x7f7d28108670 2024-04-26T21:34:15.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.246+0000 7f7d2dc07640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7d2810e390 con 0x7f7d28105d40 2024-04-26T21:34:15.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.246+0000 7f7d2dc07640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7d2810e690 con 0x7f7d28103410 2024-04-26T21:34:15.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.246+0000 7f7d27fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d28108670 0x7f7d281a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.246+0000 7f7d27fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d28108670 0x7f7d281a01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40372/0 (socket says 172.21.15.46:40372) 2024-04-26T21:34:15.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.246+0000 7f7d27fff640 1 -- 172.21.15.46:0/2446965378 learned_addr learned my addr 172.21.15.46:0/2446965378 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.247+0000 7f7d277fe640 1 --2- 172.21.15.46:0/2446965378 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d28103410 0x7f7d28198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.247+0000 7f7d26ffd640 1 --2- 172.21.15.46:0/2446965378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d28105d40 0x7f7d28199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.247+0000 7f7d27fff640 1 -- 172.21.15.46:0/2446965378 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d28103410 msgr2=0x7f7d28198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.247+0000 7f7d27fff640 1 --2- 172.21.15.46:0/2446965378 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d28103410 0x7f7d28198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.247+0000 7f7d27fff640 1 -- 172.21.15.46:0/2446965378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d28105d40 msgr2=0x7f7d28199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.247+0000 7f7d27fff640 1 --2- 172.21.15.46:0/2446965378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d28105d40 0x7f7d28199180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.247+0000 7f7d27fff640 1 -- 172.21.15.46:0/2446965378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d281a0850 con 0x7f7d28108670 2024-04-26T21:34:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.247+0000 7f7d27fff640 1 --2- 172.21.15.46:0/2446965378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d28108670 0x7f7d281a01b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000e970 tx=0x7f7d2000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.247+0000 7f7d24ff9640 1 -- 172.21.15.46:0/2446965378 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2000ccc0 con 0x7f7d28108670 2024-04-26T21:34:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.247+0000 7f7d24ff9640 1 -- 172.21.15.46:0/2446965378 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d2000ce60 con 0x7f7d28108670 2024-04-26T21:34:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.247+0000 7f7d2dc07640 1 -- 172.21.15.46:0/2446965378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d281a0b40 con 0x7f7d28108670 2024-04-26T21:34:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.247+0000 7f7d24ff9640 1 -- 172.21.15.46:0/2446965378 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d20010430 con 0x7f7d28108670 2024-04-26T21:34:15.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.247+0000 7f7d2dc07640 1 -- 172.21.15.46:0/2446965378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d281a1040 con 0x7f7d28108670 2024-04-26T21:34:15.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.249+0000 7f7d24ff9640 1 -- 172.21.15.46:0/2446965378 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7d20004ff0 con 0x7f7d28108670 2024-04-26T21:34:15.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.249+0000 7f7d309f7740 1 Processor -- start 2024-04-26T21:34:15.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.249+0000 7f7d309f7740 1 -- start start 2024-04-26T21:34:15.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.250+0000 7f7d309f7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d4979a690 0x558d4979aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.250+0000 7f7d309f7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558d4979b080 0x558d4979fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.250+0000 7f7d309f7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d497a01f0 0x558d497a2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.250+0000 7f7d309f7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558d496ff740 con 0x558d497a01f0 2024-04-26T21:34:15.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.250+0000 7f7d309f7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x558d4969b3a0 con 0x558d4979b080 2024-04-26T21:34:15.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.250+0000 7f7d309f7740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558d496fb700 con 0x558d4979a690 2024-04-26T21:34:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.250+0000 7f7d26ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d497a01f0 0x558d497a2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.250+0000 7f7d26ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d497a01f0 0x558d497a2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40378/0 (socket says 172.21.15.46:40378) 2024-04-26T21:34:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.250+0000 7f7d26ffd640 1 -- 172.21.15.46:0/1924385939 learned_addr learned my addr 172.21.15.46:0/1924385939 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.250+0000 7f7d277fe640 1 --2- 172.21.15.46:0/1924385939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558d4979b080 0x558d4979fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.250+0000 7f7d26ffd640 1 --2- 172.21.15.46:0/1924385939 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d4979a690 0x558d4979aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.250+0000 7f7d24ff9640 1 --2- 172.21.15.46:0/2446965378 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7d100a3c40 0x7f7d100a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.250+0000 7f7d26ffd640 1 -- 172.21.15.46:0/1924385939 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d4979a690 msgr2=0x558d4979aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.250+0000 7f7d26ffd640 1 --2- 172.21.15.46:0/1924385939 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d4979a690 0x558d4979aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.250+0000 7f7d26ffd640 1 -- 172.21.15.46:0/1924385939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558d4979b080 msgr2=0x558d4979fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.250+0000 7f7d26ffd640 1 --2- 172.21.15.46:0/1924385939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558d4979b080 0x558d4979fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.250+0000 7f7d26ffd640 1 -- 172.21.15.46:0/1924385939 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d496b7750 con 0x558d497a01f0 2024-04-26T21:34:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.250+0000 7f7d27fff640 1 --2- 172.21.15.46:0/2446965378 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7d100a3c40 0x7f7d100a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.250+0000 7f7d24ff9640 1 -- 172.21.15.46:0/2446965378 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f7d200d0b30 con 0x7f7d28108670 2024-04-26T21:34:15.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.250+0000 7f7d26ffd640 1 --2- 172.21.15.46:0/1924385939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d497a01f0 0x558d497a2680 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c0058c0 tx=0x7f7d1c00c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfc690dbab899790 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.251+0000 7f7d0dffb640 1 -- 172.21.15.46:0/1924385939 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c019030 con 0x558d497a01f0 2024-04-26T21:34:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.251+0000 7f7d0dffb640 1 -- 172.21.15.46:0/1924385939 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d1c0119d0 con 0x558d497a01f0 2024-04-26T21:34:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.251+0000 7f7d0dffb640 1 -- 172.21.15.46:0/1924385939 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c00fa50 con 0x558d497a01f0 2024-04-26T21:34:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.251+0000 7f7d309f7740 1 -- 172.21.15.46:0/1924385939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d497a01f0 msgr2=0x558d497a2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.251+0000 7f7d309f7740 1 --2- 172.21.15.46:0/1924385939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d497a01f0 0x558d497a2680 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c0058c0 tx=0x7f7d1c00c6b0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.251+0000 7f7d27fff640 1 --2- 172.21.15.46:0/2446965378 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7d100a3c40 0x7f7d100a6100 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7d2009c9e0 tx=0x7f7d2006e6e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.251+0000 7f7d309f7740 1 -- 172.21.15.46:0/1924385939 shutdown_connections 2024-04-26T21:34:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.251+0000 7f7d309f7740 1 --2- 172.21.15.46:0/1924385939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d497a01f0 0x558d497a2680 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.251+0000 7f7d309f7740 1 --2- 172.21.15.46:0/1924385939 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558d4979b080 0x558d4979fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.251+0000 7f7d309f7740 1 --2- 172.21.15.46:0/1924385939 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d4979a690 0x558d4979aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.251+0000 7f7d309f7740 1 -- 172.21.15.46:0/1924385939 >> 172.21.15.46:0/1924385939 conn(0x558d49795d00 msgr2=0x558d49796100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.251+0000 7f7d309f7740 1 -- 172.21.15.46:0/1924385939 shutdown_connections 2024-04-26T21:34:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.251+0000 7f7d309f7740 1 -- 172.21.15.46:0/1924385939 wait complete. 2024-04-26T21:34:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.252+0000 7f7d309f7740 1 -- start start 2024-04-26T21:34:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.252+0000 7f7d309f7740 1 Processor -- start 2024-04-26T21:34:15.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.252+0000 7f7d309f7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d4979a690 0x558d497a4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.252+0000 7f7d309f7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d4979b080 0x558d497a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.252+0000 7f7d309f7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558d497a01f0 0x558d497abfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.252+0000 7f7d309f7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558d497a31a0 con 0x558d4979b080 2024-04-26T21:34:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.252+0000 7f7d309f7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x558d49686fe0 con 0x558d497a01f0 2024-04-26T21:34:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.252+0000 7f7d309f7740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558d497a3320 con 0x558d4979a690 2024-04-26T21:34:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.252+0000 7f7d309f7740 10 client.0 _ll_register_callbacks cb 0x558d4969d150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:34:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.252+0000 7f7d277fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d4979b080 0x558d497a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.252+0000 7f7d277fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d4979b080 0x558d497a4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40388/0 (socket says 172.21.15.46:40388) 2024-04-26T21:34:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.252+0000 7f7d277fe640 1 -- 172.21.15.46:0/2574554996 learned_addr learned my addr 172.21.15.46:0/2574554996 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.252+0000 7f7d26ffd640 1 --2- 172.21.15.46:0/2574554996 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d4979a690 0x558d497a4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.252+0000 7f7d277fe640 1 -- 172.21.15.46:0/2574554996 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d4979a690 msgr2=0x558d497a4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.252+0000 7f7d277fe640 1 --2- 172.21.15.46:0/2574554996 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d4979a690 0x558d497a4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.252+0000 7f7d277fe640 1 -- 172.21.15.46:0/2574554996 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558d497a01f0 msgr2=0x558d497abfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.252+0000 7f7d26ffd640 1 --2- 172.21.15.46:0/2574554996 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558d497a01f0 0x558d497abfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.253+0000 7f7d277fe640 1 --2- 172.21.15.46:0/2574554996 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558d497a01f0 0x558d497abfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.253+0000 7f7d277fe640 1 -- 172.21.15.46:0/2574554996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d494112d0 con 0x558d4979b080 2024-04-26T21:34:15.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.253+0000 7f7d26ffd640 1 --2- 172.21.15.46:0/2574554996 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558d497a01f0 0x558d497abfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:34:15.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.253+0000 7f7d277fe640 1 --2- 172.21.15.46:0/2574554996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d4979b080 0x558d497a4fa0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7d18005e30 tx=0x7f7d18009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c83e02571d229009 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:15.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.253+0000 7f7d06ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-26T21:34:15.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.253+0000 7f7d06ffd640 1 -- 172.21.15.46:0/2574554996 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d18003b80 con 0x558d4979b080 2024-04-26T21:34:15.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.253+0000 7f7d06ffd640 1 -- 172.21.15.46:0/2574554996 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f7d18003d20 con 0x558d4979b080 2024-04-26T21:34:15.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.253+0000 7f7d309f7740 1 -- 172.21.15.46:0/2574554996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d1c0119d0 con 0x558d4979b080 2024-04-26T21:34:15.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.253+0000 7f7d06ffd640 1 -- 172.21.15.46:0/2574554996 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f7d18004040 con 0x558d4979b080 2024-04-26T21:34:15.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.253+0000 7f7d309f7740 10 client.15078 fetch_fsmap learned FSMap version 22 2024-04-26T21:34:15.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.253+0000 7f7d309f7740 1 -- 172.21.15.46:0/2574554996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x558d494db6b0 con 0x558d4979b080 2024-04-26T21:34:15.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.253+0000 7f7d06ffd640 1 -- 172.21.15.46:0/2574554996 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d1c0119d0 con 0x558d4979b080 2024-04-26T21:34:15.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.253+0000 7f7d06ffd640 1 -- 172.21.15.46:0/2574554996 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f7d1802fa90 con 0x558d4979b080 2024-04-26T21:34:15.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.253+0000 7f7d309f7740 10 client.15078 fetch_fsmap finished waiting for FSMap version 22 2024-04-26T21:34:15.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.253+0000 7f7d309f7740 10 client.15078 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T21:34:15.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.253+0000 7f7d309f7740 20 client.15078 populate_metadata read hostname 'smithi046' 2024-04-26T21:34:15.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.253+0000 7f7d309f7740 1 --2- 172.21.15.46:0/2574554996 >> [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] conn(0x558d497b1550 0x558d497b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.253+0000 7f7d309f7740 4 client.15078 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:34:15.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.253+0000 7f7d309f7740 1 -- 172.21.15.46:0/2574554996 --> [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d4970d350 con 0x558d497b1550 2024-04-26T21:34:15.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.254+0000 7f7d26ffd640 1 --2- 172.21.15.46:0/2574554996 >> [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] conn(0x558d497b1550 0x558d497b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.254+0000 7f7d26ffd640 1 --2- 172.21.15.46:0/2574554996 >> [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] conn(0x558d497b1550 0x558d497b29c0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=662237690840705c server_cookie=2954bd604f678b71 in_seq=0 out_seq=0 2024-04-26T21:34:15.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.254+0000 7f7d06ffd640 10 client.15078 ms_handle_connect on v2:172.21.15.50:6836/3189620395 2024-04-26T21:34:15.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.255+0000 7f7d06ffd640 1 -- 172.21.15.46:0/2574554996 <== mds.2 v2:172.21.15.50:6836/3189620395 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558d4970d350 con 0x558d497b1550 2024-04-26T21:34:15.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.255+0000 7f7d06ffd640 10 client.15078 handle_command_reply: tid=0 2024-04-26T21:34:15.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.255+0000 7f7d309f7740 1 client.15078 shutdown 2024-04-26T21:34:15.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.255+0000 7f7d309f7740 1 -- 172.21.15.46:0/2574554996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d4979b080 msgr2=0x558d497a4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.255+0000 7f7d309f7740 1 --2- 172.21.15.46:0/2574554996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d4979b080 0x558d497a4fa0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7d18005e30 tx=0x7f7d18009c20 comp rx=0 tx=0).stop 2024-04-26T21:34:15.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.255+0000 7f7d309f7740 1 -- 172.21.15.46:0/2574554996 shutdown_connections 2024-04-26T21:34:15.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.255+0000 7f7d309f7740 1 --2- 172.21.15.46:0/2574554996 >> [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] conn(0x558d497b1550 0x558d497b29c0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.255+0000 7f7d309f7740 1 --2- 172.21.15.46:0/2574554996 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558d497a01f0 0x558d497abfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.255+0000 7f7d309f7740 1 --2- 172.21.15.46:0/2574554996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d4979b080 0x558d497a4fa0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.255+0000 7f7d06ffd640 0 client.15078 ms_handle_reset on v2:172.21.15.50:6836/3189620395 2024-04-26T21:34:15.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.255+0000 7f7d309f7740 1 --2- 172.21.15.46:0/2574554996 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d4979a690 0x558d497a4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.255+0000 7f7d309f7740 1 -- 172.21.15.46:0/2574554996 >> 172.21.15.46:0/2574554996 conn(0x558d49795d00 msgr2=0x558d49794f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:15.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.255+0000 7f7d309f7740 1 -- 172.21.15.46:0/2574554996 shutdown_connections 2024-04-26T21:34:15.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.255+0000 7f7d309f7740 1 -- 172.21.15.46:0/2574554996 wait complete. 2024-04-26T21:34:15.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.255+0000 7f7d309f7740 20 client.15078 trim_cache size 0 max 16384 2024-04-26T21:34:15.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.274+0000 7f7d309f7740 1 Processor -- start 2024-04-26T21:34:15.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.274+0000 7f7d309f7740 1 -- start start 2024-04-26T21:34:15.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.274+0000 7f7d309f7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d4979a690 0x558d49794670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.274+0000 7f7d309f7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d4979b080 0x558d497abfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.274+0000 7f7d309f7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558d497a01f0 0x558d49794bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.274+0000 7f7d309f7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558d497b0d00 con 0x558d4979a690 2024-04-26T21:34:15.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.274+0000 7f7d309f7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x558d497b13d0 con 0x558d497a01f0 2024-04-26T21:34:15.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.274+0000 7f7d309f7740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558d497b1aa0 con 0x558d4979b080 2024-04-26T21:34:15.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.274+0000 7f7d26ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d4979a690 0x558d49794670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.274+0000 7f7d26ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d4979a690 0x558d49794670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40394/0 (socket says 172.21.15.46:40394) 2024-04-26T21:34:15.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.274+0000 7f7d26ffd640 1 -- 172.21.15.46:0/3710380480 learned_addr learned my addr 172.21.15.46:0/3710380480 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:15.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.275+0000 7f7d277fe640 1 --2- 172.21.15.46:0/3710380480 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d4979b080 0x558d497abfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.275+0000 7f7d26ffd640 1 -- 172.21.15.46:0/3710380480 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d4979b080 msgr2=0x558d497abfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:15.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.275+0000 7f7d26ffd640 1 --2- 172.21.15.46:0/3710380480 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d4979b080 0x558d497abfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.275+0000 7f7d26ffd640 1 -- 172.21.15.46:0/3710380480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558d497a01f0 msgr2=0x558d49794bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:15.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.275+0000 7f7d26ffd640 1 --2- 172.21.15.46:0/3710380480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558d497a01f0 0x558d49794bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.275+0000 7f7d26ffd640 1 -- 172.21.15.46:0/3710380480 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d4979b460 con 0x558d4979a690 2024-04-26T21:34:15.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.275+0000 7f7d26ffd640 1 --2- 172.21.15.46:0/3710380480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d4979a690 0x558d49794670 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c005d00 tx=0x7f7d1c00c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2cecb4141ccae29 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:15.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.275+0000 7f7d0d7fa640 1 -- 172.21.15.46:0/3710380480 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c02d020 con 0x558d4979a690 2024-04-26T21:34:15.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.275+0000 7f7d0d7fa640 1 -- 172.21.15.46:0/3710380480 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d1c01a040 con 0x558d4979a690 2024-04-26T21:34:15.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.275+0000 7f7d0d7fa640 1 -- 172.21.15.46:0/3710380480 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c019040 con 0x558d4979a690 2024-04-26T21:34:15.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.276+0000 7f7d309f7740 1 -- 172.21.15.46:0/3710380480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d4979a690 msgr2=0x558d49794670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.276+0000 7f7d309f7740 1 --2- 172.21.15.46:0/3710380480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d4979a690 0x558d49794670 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c005d00 tx=0x7f7d1c00c6b0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.276+0000 7f7d309f7740 1 -- 172.21.15.46:0/3710380480 shutdown_connections 2024-04-26T21:34:15.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.276+0000 7f7d309f7740 1 --2- 172.21.15.46:0/3710380480 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558d497a01f0 0x558d49794bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.276+0000 7f7d309f7740 1 --2- 172.21.15.46:0/3710380480 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d4979b080 0x558d497abfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.276+0000 7f7d309f7740 1 --2- 172.21.15.46:0/3710380480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d4979a690 0x558d49794670 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.276+0000 7f7d309f7740 1 -- 172.21.15.46:0/3710380480 >> 172.21.15.46:0/3710380480 conn(0x558d49795d00 msgr2=0x558d4979c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:15.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.276+0000 7f7d309f7740 1 -- 172.21.15.46:0/3710380480 shutdown_connections 2024-04-26T21:34:15.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.276+0000 7f7d309f7740 1 -- 172.21.15.46:0/3710380480 wait complete. 2024-04-26T21:34:15.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.276+0000 7f7d309f7740 1 -- start start 2024-04-26T21:34:15.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.276+0000 7f7d309f7740 1 Processor -- start 2024-04-26T21:34:15.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.276+0000 7f7d309f7740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d4979a690 0x558d497a7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.276+0000 7f7d309f7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d4979b080 0x558d497a82a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.276+0000 7f7d309f7740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558d497a01f0 0x558d497ab270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.276+0000 7f7d309f7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x558d497b2840 con 0x558d4979b080 2024-04-26T21:34:15.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.276+0000 7f7d309f7740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x558d497b2170 con 0x558d497a01f0 2024-04-26T21:34:15.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.276+0000 7f7d309f7740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x558d497b2f10 con 0x558d4979a690 2024-04-26T21:34:15.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.277+0000 7f7d309f7740 10 client.0 _ll_register_callbacks cb 0x7f7d18009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:34:15.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.277+0000 7f7d277fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d4979b080 0x558d497a82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.277+0000 7f7d277fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d4979b080 0x558d497a82a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40406/0 (socket says 172.21.15.46:40406) 2024-04-26T21:34:15.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.277+0000 7f7d277fe640 1 -- 172.21.15.46:0/3795301078 learned_addr learned my addr 172.21.15.46:0/3795301078 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:15.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.277+0000 7f7d26ffd640 1 --2- 172.21.15.46:0/3795301078 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558d497a01f0 0x558d497ab270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.277+0000 7f7d26ffd640 1 --2- 172.21.15.46:0/3795301078 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d4979a690 0x558d497a7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.277+0000 7f7d277fe640 1 -- 172.21.15.46:0/3795301078 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d4979a690 msgr2=0x558d497a7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:15.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.277+0000 7f7d277fe640 1 --2- 172.21.15.46:0/3795301078 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d4979a690 0x558d497a7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.277+0000 7f7d277fe640 1 -- 172.21.15.46:0/3795301078 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558d497a01f0 msgr2=0x558d497ab270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:15.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.277+0000 7f7d277fe640 1 --2- 172.21.15.46:0/3795301078 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558d497a01f0 0x558d497ab270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.277+0000 7f7d277fe640 1 -- 172.21.15.46:0/3795301078 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d497ab860 con 0x558d4979b080 2024-04-26T21:34:15.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.277+0000 7f7d277fe640 1 --2- 172.21.15.46:0/3795301078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d4979b080 0x558d497a82a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7d18002230 tx=0x7f7d180308d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4874798c8bdf5908 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:15.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.277+0000 7f7d06ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-26T21:34:15.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.277+0000 7f7d06ffd640 1 -- 172.21.15.46:0/3795301078 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d18054030 con 0x558d4979b080 2024-04-26T21:34:15.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.277+0000 7f7d06ffd640 1 -- 172.21.15.46:0/3795301078 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f7d18042040 con 0x558d4979b080 2024-04-26T21:34:15.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.277+0000 7f7d309f7740 1 -- 172.21.15.46:0/3795301078 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d1c01a040 con 0x558d4979b080 2024-04-26T21:34:15.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.277+0000 7f7d06ffd640 1 -- 172.21.15.46:0/3795301078 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f7d18004b40 con 0x558d4979b080 2024-04-26T21:34:15.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.277+0000 7f7d309f7740 10 client.15084 fetch_fsmap learned FSMap version 22 2024-04-26T21:34:15.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.277+0000 7f7d309f7740 1 -- 172.21.15.46:0/3795301078 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x558d49817b80 con 0x558d4979b080 2024-04-26T21:34:15.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.277+0000 7f7d06ffd640 1 -- 172.21.15.46:0/3795301078 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d1c01a040 con 0x558d4979b080 2024-04-26T21:34:15.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.278+0000 7f7d06ffd640 1 -- 172.21.15.46:0/3795301078 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f7d180044e0 con 0x558d4979b080 2024-04-26T21:34:15.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.278+0000 7f7d309f7740 10 client.15084 fetch_fsmap finished waiting for FSMap version 22 2024-04-26T21:34:15.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.278+0000 7f7d309f7740 10 client.15084 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T21:34:15.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.278+0000 7f7d309f7740 20 client.15084 populate_metadata read hostname 'smithi046' 2024-04-26T21:34:15.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.278+0000 7f7d309f7740 1 --2- 172.21.15.46:0/3795301078 >> [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] conn(0x558d4981c770 0x558d49819280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.278+0000 7f7d309f7740 4 client.15084 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T21:34:15.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.278+0000 7f7d309f7740 1 -- 172.21.15.46:0/3795301078 --> [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] -- command(tid 0: {"prefix": "status"}) -- 0x558d497071c0 con 0x558d4981c770 2024-04-26T21:34:15.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.278+0000 7f7d26ffd640 1 --2- 172.21.15.46:0/3795301078 >> [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] conn(0x558d4981c770 0x558d49819280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.278+0000 7f7d06ffd640 10 client.15084 ms_handle_connect on v2:172.21.15.50:6836/3189620395 2024-04-26T21:34:15.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.278+0000 7f7d26ffd640 1 --2- 172.21.15.46:0/3795301078 >> [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] conn(0x558d4981c770 0x558d49819280 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c2b28b09a4c62a7 server_cookie=17a390380814b216 in_seq=0 out_seq=0 2024-04-26T21:34:15.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.279+0000 7f7d06ffd640 1 -- 172.21.15.46:0/3795301078 <== mds.2 v2:172.21.15.50:6836/3189620395 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x558d497071c0 con 0x558d4981c770 2024-04-26T21:34:15.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.279+0000 7f7d06ffd640 10 client.15084 handle_command_reply: tid=0 2024-04-26T21:34:15.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.279+0000 7f7d309f7740 1 client.15084 shutdown 2024-04-26T21:34:15.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.279+0000 7f7d309f7740 1 -- 172.21.15.46:0/3795301078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d4979b080 msgr2=0x558d497a82a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.279+0000 7f7d309f7740 1 --2- 172.21.15.46:0/3795301078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d4979b080 0x558d497a82a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7d18002230 tx=0x7f7d180308d0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.279+0000 7f7d309f7740 1 -- 172.21.15.46:0/3795301078 shutdown_connections 2024-04-26T21:34:15.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.279+0000 7f7d309f7740 1 --2- 172.21.15.46:0/3795301078 >> [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] conn(0x558d4981c770 0x558d49819280 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.279+0000 7f7d309f7740 1 --2- 172.21.15.46:0/3795301078 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x558d497a01f0 0x558d497ab270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.279+0000 7f7d06ffd640 0 client.15084 ms_handle_reset on v2:172.21.15.50:6836/3189620395 2024-04-26T21:34:15.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.279+0000 7f7d309f7740 1 --2- 172.21.15.46:0/3795301078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558d4979b080 0x558d497a82a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.279+0000 7f7d309f7740 1 --2- 172.21.15.46:0/3795301078 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x558d4979a690 0x558d497a7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.279+0000 7f7d309f7740 1 -- 172.21.15.46:0/3795301078 >> 172.21.15.46:0/3795301078 conn(0x558d49795d00 msgr2=0x558d49794670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:15.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.279+0000 7f7d309f7740 1 -- 172.21.15.46:0/3795301078 shutdown_connections 2024-04-26T21:34:15.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.279+0000 7f7d309f7740 1 -- 172.21.15.46:0/3795301078 wait complete. 2024-04-26T21:34:15.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.279+0000 7f7d309f7740 20 client.15084 trim_cache size 0 max 16384 2024-04-26T21:34:15.286 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-26T21:34:15.286 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36c7013a-0413-11ef-bc93-c7b262605968", 2024-04-26T21:34:15.286 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 2, 2024-04-26T21:34:15.286 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 24509, 2024-04-26T21:34:15.287 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-04-26T21:34:15.287 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-04-26T21:34:15.287 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-04-26T21:34:15.287 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 86.395865916000005, 2024-04-26T21:34:15.287 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 22, 2024-04-26T21:34:15.287 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 87, 2024-04-26T21:34:15.287 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 85, 2024-04-26T21:34:15.287 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 107.962513457 2024-04-26T21:34:15.287 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-26T21:34:15.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.279+0000 7f7d2dc07640 1 -- 172.21.15.46:0/2446965378 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7d100a3c40 msgr2=0x7f7d100a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.279+0000 7f7d2dc07640 1 --2- 172.21.15.46:0/2446965378 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7d100a3c40 0x7f7d100a6100 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7d2009c9e0 tx=0x7f7d2006e6e0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.279+0000 7f7d2dc07640 1 -- 172.21.15.46:0/2446965378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d28108670 msgr2=0x7f7d281a01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.279+0000 7f7d2dc07640 1 --2- 172.21.15.46:0/2446965378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d28108670 0x7f7d281a01b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000e970 tx=0x7f7d2000ed70 comp rx=0 tx=0).stop 2024-04-26T21:34:15.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.279+0000 7f7d2dc07640 1 -- 172.21.15.46:0/2446965378 shutdown_connections 2024-04-26T21:34:15.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.279+0000 7f7d2dc07640 1 --2- 172.21.15.46:0/2446965378 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7d100a3c40 0x7f7d100a6100 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.280+0000 7f7d2dc07640 1 --2- 172.21.15.46:0/2446965378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d28108670 0x7f7d281a01b0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.280+0000 7f7d2dc07640 1 --2- 172.21.15.46:0/2446965378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d28105d40 0x7f7d28199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.280+0000 7f7d2dc07640 1 --2- 172.21.15.46:0/2446965378 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7d28103410 0x7f7d28198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.280+0000 7f7d2dc07640 1 -- 172.21.15.46:0/2446965378 >> 172.21.15.46:0/2446965378 conn(0x7f7d280fce20 msgr2=0x7f7d28103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:15.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.280+0000 7f7d2dc07640 1 -- 172.21.15.46:0/2446965378 shutdown_connections 2024-04-26T21:34:15.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.280+0000 7f7d2dc07640 1 -- 172.21.15.46:0/2446965378 wait complete. 2024-04-26T21:34:15.291 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-26T21:34:15.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.393+0000 7f2b64cfc640 1 -- 172.21.15.46:0/1002252760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b60103470 msgr2=0x7f2b60105860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.393+0000 7f2b64cfc640 1 --2- 172.21.15.46:0/1002252760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b60103470 0x7f2b60105860 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2b50009900 tx=0x7f2b5002a410 comp rx=0 tx=0).stop 2024-04-26T21:34:15.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.393+0000 7f2b64cfc640 1 -- 172.21.15.46:0/1002252760 shutdown_connections 2024-04-26T21:34:15.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.393+0000 7f2b64cfc640 1 --2- 172.21.15.46:0/1002252760 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b601086d0 0x7f2b6010aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.393+0000 7f2b64cfc640 1 --2- 172.21.15.46:0/1002252760 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b60105da0 0x7f2b60108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.393+0000 7f2b64cfc640 1 --2- 172.21.15.46:0/1002252760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b60103470 0x7f2b60105860 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.393+0000 7f2b64cfc640 1 -- 172.21.15.46:0/1002252760 >> 172.21.15.46:0/1002252760 conn(0x7f2b600fce20 msgr2=0x7f2b600ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:15.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.393+0000 7f2b64cfc640 1 -- 172.21.15.46:0/1002252760 shutdown_connections 2024-04-26T21:34:15.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.393+0000 7f2b64cfc640 1 -- 172.21.15.46:0/1002252760 wait complete. 2024-04-26T21:34:15.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.394+0000 7f2b64cfc640 1 Processor -- start 2024-04-26T21:34:15.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.394+0000 7f2b64cfc640 1 -- start start 2024-04-26T21:34:15.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.394+0000 7f2b64cfc640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b60103470 0x7f2b60198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.394+0000 7f2b64cfc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b60105da0 0x7f2b60199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.394+0000 7f2b64cfc640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b601086d0 0x7f2b601a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.394+0000 7f2b64cfc640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6010e500 con 0x7f2b60105da0 2024-04-26T21:34:15.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.394+0000 7f2b64cfc640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6010e380 con 0x7f2b601086d0 2024-04-26T21:34:15.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.394+0000 7f2b64cfc640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6010e680 con 0x7f2b60103470 2024-04-26T21:34:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.394+0000 7f2b57fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b60105da0 0x7f2b60199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.394+0000 7f2b57fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b60105da0 0x7f2b60199190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40426/0 (socket says 172.21.15.46:40426) 2024-04-26T21:34:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.394+0000 7f2b57fff640 1 -- 172.21.15.46:0/2377327499 learned_addr learned my addr 172.21.15.46:0/2377327499 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.394+0000 7f2b5ed9d640 1 --2- 172.21.15.46:0/2377327499 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b601086d0 0x7f2b601a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.394+0000 7f2b5e59c640 1 --2- 172.21.15.46:0/2377327499 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b60103470 0x7f2b60198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.394+0000 7f2b57fff640 1 -- 172.21.15.46:0/2377327499 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b60103470 msgr2=0x7f2b60198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.394+0000 7f2b57fff640 1 --2- 172.21.15.46:0/2377327499 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b60103470 0x7f2b60198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.394+0000 7f2b57fff640 1 -- 172.21.15.46:0/2377327499 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b601086d0 msgr2=0x7f2b601a01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.394+0000 7f2b57fff640 1 --2- 172.21.15.46:0/2377327499 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b601086d0 0x7f2b601a01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.394+0000 7f2b57fff640 1 -- 172.21.15.46:0/2377327499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b601a0860 con 0x7f2b60105da0 2024-04-26T21:34:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.395+0000 7f2b57fff640 1 --2- 172.21.15.46:0/2377327499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b60105da0 0x7f2b60199190 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2b4800d770 tx=0x7f2b4800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.395+0000 7f2b577fe640 1 -- 172.21.15.46:0/2377327499 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b48017070 con 0x7f2b60105da0 2024-04-26T21:34:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.395+0000 7f2b64cfc640 1 -- 172.21.15.46:0/2377327499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b601a0b50 con 0x7f2b60105da0 2024-04-26T21:34:15.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.395+0000 7f2b577fe640 1 -- 172.21.15.46:0/2377327499 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b48009c30 con 0x7f2b60105da0 2024-04-26T21:34:15.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.395+0000 7f2b577fe640 1 -- 172.21.15.46:0/2377327499 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b4800f620 con 0x7f2b60105da0 2024-04-26T21:34:15.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.395+0000 7f2b64cfc640 1 -- 172.21.15.46:0/2377327499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b601a10f0 con 0x7f2b60105da0 2024-04-26T21:34:15.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.397+0000 7f2b577fe640 1 -- 172.21.15.46:0/2377327499 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2b4801a070 con 0x7f2b60105da0 2024-04-26T21:34:15.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.397+0000 7f2b67a11740 1 Processor -- start 2024-04-26T21:34:15.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.398+0000 7f2b67a11740 1 -- start start 2024-04-26T21:34:15.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.398+0000 7f2b67a11740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600841a3aa0 0x5600841a3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.398+0000 7f2b67a11740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600841a4490 0x5600841a8f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.398+0000 7f2b67a11740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600841a9600 0x5600841aba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.398+0000 7f2b67a11740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x560084104740 con 0x5600841a9600 2024-04-26T21:34:15.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.398+0000 7f2b67a11740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5600840a03a0 con 0x5600841a4490 2024-04-26T21:34:15.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.398+0000 7f2b67a11740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x560084100700 con 0x5600841a3aa0 2024-04-26T21:34:15.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.398+0000 7f2b5e59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600841a4490 0x5600841a8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.398+0000 7f2b57fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600841a9600 0x5600841aba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.398+0000 7f2b5e59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600841a4490 0x5600841a8f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.46:56226/0 (socket says 172.21.15.46:56226) 2024-04-26T21:34:15.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.398+0000 7f2b5e59c640 1 -- 172.21.15.46:0/3617280849 learned_addr learned my addr 172.21.15.46:0/3617280849 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:15.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.398+0000 7f2b577fe640 1 --2- 172.21.15.46:0/2377327499 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f2b4c0a3c40 0x7f2b4c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.398+0000 7f2b5ed9d640 1 --2- 172.21.15.46:0/3617280849 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600841a3aa0 0x5600841a3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.398+0000 7f2b5e59c640 1 -- 172.21.15.46:0/3617280849 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600841a3aa0 msgr2=0x5600841a3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:15.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.398+0000 7f2b5e59c640 1 --2- 172.21.15.46:0/3617280849 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600841a3aa0 0x5600841a3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.398+0000 7f2b5ed9d640 1 --2- 172.21.15.46:0/2377327499 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f2b4c0a3c40 0x7f2b4c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.398+0000 7f2b5e59c640 1 -- 172.21.15.46:0/3617280849 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600841a9600 msgr2=0x5600841aba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:15.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.398+0000 7f2b5e59c640 1 --2- 172.21.15.46:0/3617280849 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600841a9600 0x5600841aba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.398+0000 7f2b5e59c640 1 -- 172.21.15.46:0/3617280849 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600840bc750 con 0x5600841a4490 2024-04-26T21:34:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.398+0000 7f2b577fe640 1 -- 172.21.15.46:0/2377327499 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f2b480d0240 con 0x7f2b60105da0 2024-04-26T21:34:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.398+0000 7f2b5e59c640 1 --2- 172.21.15.46:0/3617280849 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600841a4490 0x5600841a8f90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2b50000c00 tx=0x7f2b50004160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e929e0eed438d1be server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.399+0000 7f2b5ed9d640 1 --2- 172.21.15.46:0/2377327499 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f2b4c0a3c40 0x7f2b4c0a6100 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2b58009fd0 tx=0x7f2b58005bd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.399+0000 7f2b54ff9640 1 -- 172.21.15.46:0/3617280849 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b50035030 con 0x5600841a4490 2024-04-26T21:34:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.399+0000 7f2b54ff9640 1 -- 172.21.15.46:0/3617280849 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b500054f0 con 0x5600841a4490 2024-04-26T21:34:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.399+0000 7f2b54ff9640 1 -- 172.21.15.46:0/3617280849 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b50003850 con 0x5600841a4490 2024-04-26T21:34:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.399+0000 7f2b67a11740 1 -- 172.21.15.46:0/3617280849 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600841a4490 msgr2=0x5600841a8f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.399+0000 7f2b67a11740 1 --2- 172.21.15.46:0/3617280849 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600841a4490 0x5600841a8f90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2b50000c00 tx=0x7f2b50004160 comp rx=0 tx=0).stop 2024-04-26T21:34:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.399+0000 7f2b67a11740 1 -- 172.21.15.46:0/3617280849 shutdown_connections 2024-04-26T21:34:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.399+0000 7f2b67a11740 1 --2- 172.21.15.46:0/3617280849 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600841a9600 0x5600841aba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.399+0000 7f2b67a11740 1 --2- 172.21.15.46:0/3617280849 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600841a4490 0x5600841a8f90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.399+0000 7f2b67a11740 1 --2- 172.21.15.46:0/3617280849 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600841a3aa0 0x5600841a3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.399+0000 7f2b67a11740 1 -- 172.21.15.46:0/3617280849 >> 172.21.15.46:0/3617280849 conn(0x56008419ad00 msgr2=0x56008419b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.400+0000 7f2b67a11740 1 -- 172.21.15.46:0/3617280849 shutdown_connections 2024-04-26T21:34:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.400+0000 7f2b67a11740 1 -- 172.21.15.46:0/3617280849 wait complete. 2024-04-26T21:34:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.400+0000 7f2b67a11740 1 -- start start 2024-04-26T21:34:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.400+0000 7f2b67a11740 1 Processor -- start 2024-04-26T21:34:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.400+0000 7f2b67a11740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600841a3aa0 0x5600841adef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.400+0000 7f2b67a11740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600841a4490 0x5600841ae430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.400+0000 7f2b67a11740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600841a9600 0x5600841b5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.400+0000 7f2b67a11740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5600841ac550 con 0x5600841a4490 2024-04-26T21:34:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.400+0000 7f2b67a11740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x56008408bfe0 con 0x5600841a9600 2024-04-26T21:34:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.400+0000 7f2b67a11740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5600841ac6d0 con 0x5600841a3aa0 2024-04-26T21:34:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.400+0000 7f2b67a11740 10 client.0 _ll_register_callbacks cb 0x5600840a2150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:34:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.400+0000 7f2b57fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600841a4490 0x5600841ae430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.400+0000 7f2b57fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600841a4490 0x5600841ae430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40432/0 (socket says 172.21.15.46:40432) 2024-04-26T21:34:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.400+0000 7f2b57fff640 1 -- 172.21.15.46:0/1718188419 learned_addr learned my addr 172.21.15.46:0/1718188419 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.400+0000 7f2b5e59c640 1 --2- 172.21.15.46:0/1718188419 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600841a3aa0 0x5600841adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.400+0000 7f2b5ed9d640 1 --2- 172.21.15.46:0/1718188419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600841a9600 0x5600841b5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.400+0000 7f2b57fff640 1 -- 172.21.15.46:0/1718188419 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600841a3aa0 msgr2=0x5600841adef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.400+0000 7f2b57fff640 1 --2- 172.21.15.46:0/1718188419 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600841a3aa0 0x5600841adef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.400+0000 7f2b57fff640 1 -- 172.21.15.46:0/1718188419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600841a9600 msgr2=0x5600841b5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.400+0000 7f2b57fff640 1 --2- 172.21.15.46:0/1718188419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600841a9600 0x5600841b5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.400+0000 7f2b57fff640 1 -- 172.21.15.46:0/1718188419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560083e162d0 con 0x5600841a4490 2024-04-26T21:34:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.400+0000 7f2b5e59c640 1 --2- 172.21.15.46:0/1718188419 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600841a3aa0 0x5600841adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:34:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.400+0000 7f2b5ed9d640 1 --2- 172.21.15.46:0/1718188419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600841a9600 0x5600841b5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:34:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.401+0000 7f2b57fff640 1 --2- 172.21.15.46:0/1718188419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600841a4490 0x5600841ae430 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2b480cdbf0 tx=0x7f2b48010c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d3ffd2bf30e7b70 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.401+0000 7f2b45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-26T21:34:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.401+0000 7f2b45ffb640 1 -- 172.21.15.46:0/1718188419 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b48021040 con 0x5600841a4490 2024-04-26T21:34:15.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.401+0000 7f2b67a11740 1 -- 172.21.15.46:0/1718188419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b500054f0 con 0x5600841a4490 2024-04-26T21:34:15.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.401+0000 7f2b45ffb640 1 -- 172.21.15.46:0/1718188419 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f2b48098970 con 0x5600841a4490 2024-04-26T21:34:15.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.401+0000 7f2b45ffb640 1 -- 172.21.15.46:0/1718188419 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f2b4809bcf0 con 0x5600841a4490 2024-04-26T21:34:15.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.401+0000 7f2b67a11740 10 client.15096 fetch_fsmap learned FSMap version 22 2024-04-26T21:34:15.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.401+0000 7f2b67a11740 1 -- 172.21.15.46:0/1718188419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x560083ee06b0 con 0x5600841a4490 2024-04-26T21:34:15.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.401+0000 7f2b45ffb640 1 -- 172.21.15.46:0/1718188419 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b500054f0 con 0x5600841a4490 2024-04-26T21:34:15.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.401+0000 7f2b45ffb640 1 -- 172.21.15.46:0/1718188419 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f2b480cde10 con 0x5600841a4490 2024-04-26T21:34:15.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.401+0000 7f2b67a11740 10 client.15096 fetch_fsmap finished waiting for FSMap version 22 2024-04-26T21:34:15.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.401+0000 7f2b67a11740 10 client.15096 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T21:34:15.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.401+0000 7f2b67a11740 20 client.15096 populate_metadata read hostname 'smithi046' 2024-04-26T21:34:15.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.401+0000 7f2b67a11740 1 --2- 172.21.15.46:0/1718188419 >> [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] conn(0x5600841ba9b0 0x5600841bbe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.401+0000 7f2b67a11740 4 client.15096 mds_command: new command op to 24526 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:34:15.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.401+0000 7f2b67a11740 1 -- 172.21.15.46:0/1718188419 --> [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560084112350 con 0x5600841ba9b0 2024-04-26T21:34:15.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.402+0000 7f2b5e59c640 1 --2- 172.21.15.46:0/1718188419 >> [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] conn(0x5600841ba9b0 0x5600841bbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.402+0000 7f2b5e59c640 1 --2- 172.21.15.46:0/1718188419 >> [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] conn(0x5600841ba9b0 0x5600841bbe20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c63d9c4154c7c99f server_cookie=81fd77b247870055 in_seq=0 out_seq=0 2024-04-26T21:34:15.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.402+0000 7f2b45ffb640 10 client.15096 ms_handle_connect on v2:172.21.15.142:6838/3089509101 2024-04-26T21:34:15.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.403+0000 7f2b45ffb640 1 -- 172.21.15.46:0/1718188419 <== mds.1 v2:172.21.15.142:6838/3089509101 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560084112350 con 0x5600841ba9b0 2024-04-26T21:34:15.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.403+0000 7f2b45ffb640 10 client.15096 handle_command_reply: tid=0 2024-04-26T21:34:15.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.403+0000 7f2b67a11740 1 client.15096 shutdown 2024-04-26T21:34:15.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.403+0000 7f2b67a11740 1 -- 172.21.15.46:0/1718188419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600841a4490 msgr2=0x5600841ae430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.403+0000 7f2b67a11740 1 --2- 172.21.15.46:0/1718188419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600841a4490 0x5600841ae430 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2b480cdbf0 tx=0x7f2b48010c00 comp rx=0 tx=0).stop 2024-04-26T21:34:15.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.403+0000 7f2b67a11740 1 -- 172.21.15.46:0/1718188419 shutdown_connections 2024-04-26T21:34:15.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.403+0000 7f2b67a11740 1 --2- 172.21.15.46:0/1718188419 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600841a9600 0x5600841b5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.403+0000 7f2b67a11740 1 --2- 172.21.15.46:0/1718188419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600841a4490 0x5600841ae430 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.403+0000 7f2b67a11740 1 --2- 172.21.15.46:0/1718188419 >> [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] conn(0x5600841ba9b0 0x5600841bbe20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.403+0000 7f2b67a11740 1 --2- 172.21.15.46:0/1718188419 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600841a3aa0 0x5600841adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.403+0000 7f2b45ffb640 0 client.15096 ms_handle_reset on v2:172.21.15.142:6838/3089509101 2024-04-26T21:34:15.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.403+0000 7f2b67a11740 1 -- 172.21.15.46:0/1718188419 >> 172.21.15.46:0/1718188419 conn(0x56008419ad00 msgr2=0x560084199f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:15.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.403+0000 7f2b67a11740 1 -- 172.21.15.46:0/1718188419 shutdown_connections 2024-04-26T21:34:15.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.403+0000 7f2b67a11740 1 -- 172.21.15.46:0/1718188419 wait complete. 2024-04-26T21:34:15.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.403+0000 7f2b67a11740 20 client.15096 trim_cache size 0 max 16384 2024-04-26T21:34:15.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.421+0000 7f2b67a11740 1 Processor -- start 2024-04-26T21:34:15.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.421+0000 7f2b67a11740 1 -- start start 2024-04-26T21:34:15.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.421+0000 7f2b67a11740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600841a3aa0 0x56008419b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.421+0000 7f2b67a11740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600841a4490 0x5600841b5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.421+0000 7f2b67a11740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600841a9600 0x56008419dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.421+0000 7f2b67a11740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5600841ba160 con 0x5600841a9600 2024-04-26T21:34:15.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.421+0000 7f2b67a11740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5600841ba830 con 0x5600841a3aa0 2024-04-26T21:34:15.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.421+0000 7f2b67a11740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5600841baf00 con 0x5600841a4490 2024-04-26T21:34:15.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.422+0000 7f2b5ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600841a9600 0x56008419dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.422+0000 7f2b5ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600841a9600 0x56008419dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40448/0 (socket says 172.21.15.46:40448) 2024-04-26T21:34:15.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.422+0000 7f2b5ed9d640 1 -- 172.21.15.46:0/755474531 learned_addr learned my addr 172.21.15.46:0/755474531 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:15.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.422+0000 7f2b5e59c640 1 --2- 172.21.15.46:0/755474531 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600841a3aa0 0x56008419b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.422+0000 7f2b57fff640 1 --2- 172.21.15.46:0/755474531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600841a4490 0x5600841b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.422+0000 7f2b5ed9d640 1 -- 172.21.15.46:0/755474531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600841a4490 msgr2=0x5600841b5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:15.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.422+0000 7f2b5ed9d640 1 --2- 172.21.15.46:0/755474531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600841a4490 0x5600841b5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.422+0000 7f2b5ed9d640 1 -- 172.21.15.46:0/755474531 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600841a3aa0 msgr2=0x56008419b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:15.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.422+0000 7f2b5ed9d640 1 --2- 172.21.15.46:0/755474531 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600841a3aa0 0x56008419b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.422+0000 7f2b5ed9d640 1 -- 172.21.15.46:0/755474531 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600841a8b60 con 0x5600841a9600 2024-04-26T21:34:15.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.422+0000 7f2b57fff640 1 --2- 172.21.15.46:0/755474531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600841a4490 0x5600841b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:34:15.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.422+0000 7f2b5e59c640 1 --2- 172.21.15.46:0/755474531 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600841a3aa0 0x56008419b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:34:15.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.422+0000 7f2b5ed9d640 1 --2- 172.21.15.46:0/755474531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600841a9600 0x56008419dcb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2b58010880 tx=0x7f2b58009720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa66b88af2244c2a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:15.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.422+0000 7f2b47fff640 1 -- 172.21.15.46:0/755474531 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b58010e00 con 0x5600841a9600 2024-04-26T21:34:15.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.422+0000 7f2b47fff640 1 -- 172.21.15.46:0/755474531 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b58024070 con 0x5600841a9600 2024-04-26T21:34:15.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.422+0000 7f2b47fff640 1 -- 172.21.15.46:0/755474531 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b58005480 con 0x5600841a9600 2024-04-26T21:34:15.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.423+0000 7f2b67a11740 1 -- 172.21.15.46:0/755474531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600841a9600 msgr2=0x56008419dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.423+0000 7f2b67a11740 1 --2- 172.21.15.46:0/755474531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600841a9600 0x56008419dcb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2b58010880 tx=0x7f2b58009720 comp rx=0 tx=0).stop 2024-04-26T21:34:15.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.423+0000 7f2b67a11740 1 -- 172.21.15.46:0/755474531 shutdown_connections 2024-04-26T21:34:15.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.423+0000 7f2b67a11740 1 --2- 172.21.15.46:0/755474531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600841a9600 0x56008419dcb0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.423+0000 7f2b67a11740 1 --2- 172.21.15.46:0/755474531 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600841a4490 0x5600841b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.423+0000 7f2b67a11740 1 --2- 172.21.15.46:0/755474531 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600841a3aa0 0x56008419b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.423+0000 7f2b67a11740 1 -- 172.21.15.46:0/755474531 >> 172.21.15.46:0/755474531 conn(0x56008419ad00 msgr2=0x5600841a5890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:15.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.423+0000 7f2b67a11740 1 -- 172.21.15.46:0/755474531 shutdown_connections 2024-04-26T21:34:15.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.423+0000 7f2b67a11740 1 -- 172.21.15.46:0/755474531 wait complete. 2024-04-26T21:34:15.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.423+0000 7f2b67a11740 1 -- start start 2024-04-26T21:34:15.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.423+0000 7f2b67a11740 1 Processor -- start 2024-04-26T21:34:15.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.423+0000 7f2b67a11740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600841a3aa0 0x5600841b1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.423+0000 7f2b67a11740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600841a4490 0x5600841b1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.423+0000 7f2b67a11740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600841a9600 0x5600841b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.423+0000 7f2b67a11740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5600841bbca0 con 0x5600841a9600 2024-04-26T21:34:15.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.423+0000 7f2b67a11740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5600841bb5d0 con 0x5600841a3aa0 2024-04-26T21:34:15.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.423+0000 7f2b67a11740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5600841bc370 con 0x5600841a4490 2024-04-26T21:34:15.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.423+0000 7f2b67a11740 10 client.0 _ll_register_callbacks cb 0x7f2b48010b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:34:15.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.423+0000 7f2b5ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600841a9600 0x5600841b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.423+0000 7f2b5ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600841a9600 0x5600841b48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40452/0 (socket says 172.21.15.46:40452) 2024-04-26T21:34:15.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.424+0000 7f2b5ed9d640 1 -- 172.21.15.46:0/2399081468 learned_addr learned my addr 172.21.15.46:0/2399081468 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:15.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.424+0000 7f2b5e59c640 1 --2- 172.21.15.46:0/2399081468 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600841a3aa0 0x5600841b1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.424+0000 7f2b57fff640 1 --2- 172.21.15.46:0/2399081468 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600841a4490 0x5600841b1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.424+0000 7f2b5ed9d640 1 -- 172.21.15.46:0/2399081468 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600841a4490 msgr2=0x5600841b1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:15.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.424+0000 7f2b5ed9d640 1 --2- 172.21.15.46:0/2399081468 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600841a4490 0x5600841b1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.424+0000 7f2b5ed9d640 1 -- 172.21.15.46:0/2399081468 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600841a3aa0 msgr2=0x5600841b1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:15.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.424+0000 7f2b5ed9d640 1 --2- 172.21.15.46:0/2399081468 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600841a3aa0 0x5600841b1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.424+0000 7f2b5ed9d640 1 -- 172.21.15.46:0/2399081468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600841b4e90 con 0x5600841a9600 2024-04-26T21:34:15.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.424+0000 7f2b5ed9d640 1 --2- 172.21.15.46:0/2399081468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600841a9600 0x5600841b48a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2b58010850 tx=0x7f2b58010660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71a21f0f4ec3a5c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:15.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.424+0000 7f2b45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-26T21:34:15.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.424+0000 7f2b45ffb640 1 -- 172.21.15.46:0/2399081468 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b58036060 con 0x5600841a9600 2024-04-26T21:34:15.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.424+0000 7f2b45ffb640 1 -- 172.21.15.46:0/2399081468 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f2b58019050 con 0x5600841a9600 2024-04-26T21:34:15.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.424+0000 7f2b67a11740 1 -- 172.21.15.46:0/2399081468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b58024070 con 0x5600841a9600 2024-04-26T21:34:15.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.424+0000 7f2b45ffb640 1 -- 172.21.15.46:0/2399081468 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f2b58004dd0 con 0x5600841a9600 2024-04-26T21:34:15.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.424+0000 7f2b67a11740 10 client.15102 fetch_fsmap learned FSMap version 22 2024-04-26T21:34:15.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.424+0000 7f2b67a11740 1 -- 172.21.15.46:0/2399081468 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5600842210f0 con 0x5600841a9600 2024-04-26T21:34:15.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.424+0000 7f2b45ffb640 1 -- 172.21.15.46:0/2399081468 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b58024070 con 0x5600841a9600 2024-04-26T21:34:15.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.425+0000 7f2b45ffb640 1 -- 172.21.15.46:0/2399081468 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f2b58028420 con 0x5600841a9600 2024-04-26T21:34:15.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.425+0000 7f2b67a11740 10 client.15102 fetch_fsmap finished waiting for FSMap version 22 2024-04-26T21:34:15.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.425+0000 7f2b67a11740 10 client.15102 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T21:34:15.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.425+0000 7f2b67a11740 20 client.15102 populate_metadata read hostname 'smithi046' 2024-04-26T21:34:15.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.425+0000 7f2b67a11740 1 --2- 172.21.15.46:0/2399081468 >> [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] conn(0x5600842259e0 0x5600842224f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.425+0000 7f2b67a11740 4 client.15102 mds_command: new command op to 24526 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T21:34:15.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.425+0000 7f2b67a11740 1 -- 172.21.15.46:0/2399081468 --> [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] -- command(tid 0: {"prefix": "status"}) -- 0x56008410c1c0 con 0x5600842259e0 2024-04-26T21:34:15.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.425+0000 7f2b5e59c640 1 --2- 172.21.15.46:0/2399081468 >> [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] conn(0x5600842259e0 0x5600842224f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.425+0000 7f2b5e59c640 1 --2- 172.21.15.46:0/2399081468 >> [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] conn(0x5600842259e0 0x5600842224f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9441ca9505b131b0 server_cookie=13f8c8c614a8766 in_seq=0 out_seq=0 2024-04-26T21:34:15.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.425+0000 7f2b45ffb640 10 client.15102 ms_handle_connect on v2:172.21.15.142:6838/3089509101 2024-04-26T21:34:15.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b45ffb640 1 -- 172.21.15.46:0/2399081468 <== mds.1 v2:172.21.15.142:6838/3089509101 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x56008410c1c0 con 0x5600842259e0 2024-04-26T21:34:15.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b45ffb640 10 client.15102 handle_command_reply: tid=0 2024-04-26T21:34:15.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b67a11740 1 client.15102 shutdown 2024-04-26T21:34:15.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b67a11740 1 -- 172.21.15.46:0/2399081468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600841a9600 msgr2=0x5600841b48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b67a11740 1 --2- 172.21.15.46:0/2399081468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600841a9600 0x5600841b48a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2b58010850 tx=0x7f2b58010660 comp rx=0 tx=0).stop 2024-04-26T21:34:15.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b67a11740 1 -- 172.21.15.46:0/2399081468 shutdown_connections 2024-04-26T21:34:15.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b67a11740 1 --2- 172.21.15.46:0/2399081468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5600841a9600 0x5600841b48a0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b67a11740 1 --2- 172.21.15.46:0/2399081468 >> [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] conn(0x5600842259e0 0x5600842224f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b67a11740 1 --2- 172.21.15.46:0/2399081468 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5600841a4490 0x5600841b1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b67a11740 1 --2- 172.21.15.46:0/2399081468 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5600841a3aa0 0x5600841b1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b45ffb640 0 client.15102 ms_handle_reset on v2:172.21.15.142:6838/3089509101 2024-04-26T21:34:15.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b67a11740 1 -- 172.21.15.46:0/2399081468 >> 172.21.15.46:0/2399081468 conn(0x56008419ad00 msgr2=0x560084199670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:15.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b67a11740 1 -- 172.21.15.46:0/2399081468 shutdown_connections 2024-04-26T21:34:15.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b67a11740 1 -- 172.21.15.46:0/2399081468 wait complete. 2024-04-26T21:34:15.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b67a11740 20 client.15102 trim_cache size 0 max 16384 2024-04-26T21:34:15.434 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-26T21:34:15.434 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36c7013a-0413-11ef-bc93-c7b262605968", 2024-04-26T21:34:15.434 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 1, 2024-04-26T21:34:15.435 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 24526, 2024-04-26T21:34:15.435 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-04-26T21:34:15.435 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-04-26T21:34:15.435 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-04-26T21:34:15.435 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 87.551371107999998, 2024-04-26T21:34:15.435 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 22, 2024-04-26T21:34:15.435 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 87, 2024-04-26T21:34:15.435 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 85, 2024-04-26T21:34:15.435 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 105.327612221 2024-04-26T21:34:15.435 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-26T21:34:15.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b64cfc640 1 -- 172.21.15.46:0/2377327499 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f2b4c0a3c40 msgr2=0x7f2b4c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b64cfc640 1 --2- 172.21.15.46:0/2377327499 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f2b4c0a3c40 0x7f2b4c0a6100 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2b58009fd0 tx=0x7f2b58005bd0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b64cfc640 1 -- 172.21.15.46:0/2377327499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b60105da0 msgr2=0x7f2b60199190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b64cfc640 1 --2- 172.21.15.46:0/2377327499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b60105da0 0x7f2b60199190 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2b4800d770 tx=0x7f2b4800dc30 comp rx=0 tx=0).stop 2024-04-26T21:34:15.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b64cfc640 1 -- 172.21.15.46:0/2377327499 shutdown_connections 2024-04-26T21:34:15.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b64cfc640 1 --2- 172.21.15.46:0/2377327499 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f2b4c0a3c40 0x7f2b4c0a6100 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b64cfc640 1 --2- 172.21.15.46:0/2377327499 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2b601086d0 0x7f2b601a01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b64cfc640 1 --2- 172.21.15.46:0/2377327499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b60105da0 0x7f2b60199190 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b64cfc640 1 --2- 172.21.15.46:0/2377327499 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2b60103470 0x7f2b60198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b64cfc640 1 -- 172.21.15.46:0/2377327499 >> 172.21.15.46:0/2377327499 conn(0x7f2b600fce20 msgr2=0x7f2b60103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:15.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b64cfc640 1 -- 172.21.15.46:0/2377327499 shutdown_connections 2024-04-26T21:34:15.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.426+0000 7f2b64cfc640 1 -- 172.21.15.46:0/2377327499 wait complete. 2024-04-26T21:34:15.438 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-26T21:34:15.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.540+0000 7f7de9de4640 1 -- 172.21.15.46:0/609637283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de4105d40 msgr2=0x7f7de4108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.540+0000 7f7de9de4640 1 --2- 172.21.15.46:0/609637283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de4105d40 0x7f7de4108130 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc009900 tx=0x7f7dcc02a410 comp rx=0 tx=0).stop 2024-04-26T21:34:15.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.540+0000 7f7de9de4640 1 -- 172.21.15.46:0/609637283 shutdown_connections 2024-04-26T21:34:15.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.540+0000 7f7de9de4640 1 --2- 172.21.15.46:0/609637283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7de4108670 0x7f7de410aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.540+0000 7f7de9de4640 1 --2- 172.21.15.46:0/609637283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de4105d40 0x7f7de4108130 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.540+0000 7f7de9de4640 1 --2- 172.21.15.46:0/609637283 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7de4103410 0x7f7de4105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.540+0000 7f7de9de4640 1 -- 172.21.15.46:0/609637283 >> 172.21.15.46:0/609637283 conn(0x7f7de40fce20 msgr2=0x7f7de40ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:15.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.540+0000 7f7de9de4640 1 -- 172.21.15.46:0/609637283 shutdown_connections 2024-04-26T21:34:15.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.540+0000 7f7de9de4640 1 -- 172.21.15.46:0/609637283 wait complete. 2024-04-26T21:34:15.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.540+0000 7f7de9de4640 1 Processor -- start 2024-04-26T21:34:15.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.541+0000 7f7de9de4640 1 -- start start 2024-04-26T21:34:15.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.541+0000 7f7de9de4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de4103410 0x7f7de4102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.541+0000 7f7de9de4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7de4105d40 0x7f7de41013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.541+0000 7f7de9de4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7de4108670 0x7f7de4101900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.541+0000 7f7de9de4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f7de410e4d0 con 0x7f7de4103410 2024-04-26T21:34:15.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.541+0000 7f7de9de4640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7de410e350 con 0x7f7de4105d40 2024-04-26T21:34:15.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.541+0000 7f7de9de4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f7de410e650 con 0x7f7de4108670 2024-04-26T21:34:15.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.541+0000 7f7de37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de4103410 0x7f7de4102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.541+0000 7f7de37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de4103410 0x7f7de4102d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40470/0 (socket says 172.21.15.46:40470) 2024-04-26T21:34:15.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.541+0000 7f7de37fe640 1 -- 172.21.15.46:0/2334886896 learned_addr learned my addr 172.21.15.46:0/2334886896 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:15.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.541+0000 7f7de2ffd640 1 --2- 172.21.15.46:0/2334886896 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7de4105d40 0x7f7de41013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.541+0000 7f7de3fff640 1 --2- 172.21.15.46:0/2334886896 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7de4108670 0x7f7de4101900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.541+0000 7f7de37fe640 1 -- 172.21.15.46:0/2334886896 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7de4108670 msgr2=0x7f7de4101900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.541+0000 7f7de37fe640 1 --2- 172.21.15.46:0/2334886896 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7de4108670 0x7f7de4101900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.541+0000 7f7de37fe640 1 -- 172.21.15.46:0/2334886896 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7de4105d40 msgr2=0x7f7de41013c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.541+0000 7f7de37fe640 1 --2- 172.21.15.46:0/2334886896 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7de4105d40 0x7f7de41013c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.541+0000 7f7de37fe640 1 -- 172.21.15.46:0/2334886896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7de41021c0 con 0x7f7de4103410 2024-04-26T21:34:15.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.541+0000 7f7de3fff640 1 --2- 172.21.15.46:0/2334886896 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7de4108670 0x7f7de4101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:34:15.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.541+0000 7f7de37fe640 1 --2- 172.21.15.46:0/2334886896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de4103410 0x7f7de4102d30 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7dd400d770 tx=0x7f7dd400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:15.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.541+0000 7f7de0ff9640 1 -- 172.21.15.46:0/2334886896 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4017070 con 0x7f7de4103410 2024-04-26T21:34:15.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.541+0000 7f7de0ff9640 1 -- 172.21.15.46:0/2334886896 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dd40054d0 con 0x7f7de4103410 2024-04-26T21:34:15.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.541+0000 7f7de9de4640 1 -- 172.21.15.46:0/2334886896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7de41a2d90 con 0x7f7de4103410 2024-04-26T21:34:15.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.541+0000 7f7de0ff9640 1 -- 172.21.15.46:0/2334886896 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4004c30 con 0x7f7de4103410 2024-04-26T21:34:15.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.542+0000 7f7de9de4640 1 -- 172.21.15.46:0/2334886896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7de41a31a0 con 0x7f7de4103410 2024-04-26T21:34:15.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.544+0000 7f7de0ff9640 1 -- 172.21.15.46:0/2334886896 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7dd4004e10 con 0x7f7de4103410 2024-04-26T21:34:15.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.544+0000 7f7decbd3740 1 Processor -- start 2024-04-26T21:34:15.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.544+0000 7f7decbd3740 1 -- start start 2024-04-26T21:34:15.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.544+0000 7f7decbd3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743cb690 0x5555743cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.544+0000 7f7decbd3740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555743cc080 0x5555743d0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.544+0000 7f7decbd3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555743d11f0 0x5555743d3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.544+0000 7f7decbd3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x555574330740 con 0x5555743cb690 2024-04-26T21:34:15.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.544+0000 7f7decbd3740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5555742cc3a0 con 0x5555743cc080 2024-04-26T21:34:15.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.544+0000 7f7decbd3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x55557432c700 con 0x5555743d11f0 2024-04-26T21:34:15.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.544+0000 7f7de3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743cb690 0x5555743cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.544+0000 7f7de3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743cb690 0x5555743cbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40476/0 (socket says 172.21.15.46:40476) 2024-04-26T21:34:15.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.544+0000 7f7de3fff640 1 -- 172.21.15.46:0/4181868177 learned_addr learned my addr 172.21.15.46:0/4181868177 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:15.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.544+0000 7f7de0ff9640 1 --2- 172.21.15.46:0/2334886896 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7dd00a3c40 0x7f7dd00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.544+0000 7f7de2ffd640 1 --2- 172.21.15.46:0/4181868177 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555743d11f0 0x5555743d3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.545+0000 7f7de37fe640 1 --2- 172.21.15.46:0/4181868177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555743cc080 0x5555743d0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.545+0000 7f7de3fff640 1 --2- 172.21.15.46:0/2334886896 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7dd00a3c40 0x7f7dd00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.545+0000 7f7de0ff9640 1 -- 172.21.15.46:0/2334886896 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f7dd40cff80 con 0x7f7de4103410 2024-04-26T21:34:15.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.545+0000 7f7de2ffd640 1 -- 172.21.15.46:0/4181868177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555743cc080 msgr2=0x5555743d0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:15.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.545+0000 7f7de2ffd640 1 --2- 172.21.15.46:0/4181868177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555743cc080 0x5555743d0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.545+0000 7f7de2ffd640 1 -- 172.21.15.46:0/4181868177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743cb690 msgr2=0x5555743cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.545+0000 7f7de2ffd640 1 --2- 172.21.15.46:0/4181868177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743cb690 0x5555743cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.545+0000 7f7de2ffd640 1 -- 172.21.15.46:0/4181868177 --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555742e8750 con 0x5555743d11f0 2024-04-26T21:34:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.545+0000 7f7de3fff640 1 --2- 172.21.15.46:0/4181868177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743cb690 0x5555743cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:34:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.545+0000 7f7de2ffd640 1 --2- 172.21.15.46:0/4181868177 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555743d11f0 0x5555743d3680 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc006010 tx=0x7f7dcc031b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2dd3c5b49d9be25 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.545+0000 7f7de3fff640 1 --2- 172.21.15.46:0/2334886896 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7dd00a3c40 0x7f7dd00a6100 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x5555743cc8e0 tx=0x7f7ddc004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.545+0000 7f7dd9ffb640 1 -- 172.21.15.46:0/4181868177 <== mon.1 v2:172.21.15.142:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc035030 con 0x5555743d11f0 2024-04-26T21:34:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.545+0000 7f7dd9ffb640 1 -- 172.21.15.46:0/4181868177 <== mon.1 v2:172.21.15.142:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dcc040070 con 0x5555743d11f0 2024-04-26T21:34:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.545+0000 7f7dd9ffb640 1 -- 172.21.15.46:0/4181868177 <== mon.1 v2:172.21.15.142:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc0039b0 con 0x5555743d11f0 2024-04-26T21:34:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.546+0000 7f7decbd3740 1 -- 172.21.15.46:0/4181868177 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555743d11f0 msgr2=0x5555743d3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.546+0000 7f7decbd3740 1 --2- 172.21.15.46:0/4181868177 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555743d11f0 0x5555743d3680 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc006010 tx=0x7f7dcc031b90 comp rx=0 tx=0).stop 2024-04-26T21:34:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.546+0000 7f7decbd3740 1 -- 172.21.15.46:0/4181868177 shutdown_connections 2024-04-26T21:34:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.546+0000 7f7decbd3740 1 --2- 172.21.15.46:0/4181868177 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555743d11f0 0x5555743d3680 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.546+0000 7f7decbd3740 1 --2- 172.21.15.46:0/4181868177 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555743cc080 0x5555743d0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.546+0000 7f7decbd3740 1 --2- 172.21.15.46:0/4181868177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743cb690 0x5555743cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.546+0000 7f7decbd3740 1 -- 172.21.15.46:0/4181868177 >> 172.21.15.46:0/4181868177 conn(0x5555743c6d00 msgr2=0x5555743c7100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.546+0000 7f7decbd3740 1 -- 172.21.15.46:0/4181868177 shutdown_connections 2024-04-26T21:34:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.546+0000 7f7decbd3740 1 -- 172.21.15.46:0/4181868177 wait complete. 2024-04-26T21:34:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.546+0000 7f7decbd3740 1 -- start start 2024-04-26T21:34:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.546+0000 7f7decbd3740 1 Processor -- start 2024-04-26T21:34:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.547+0000 7f7decbd3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743cb690 0x5555743d5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.547+0000 7f7decbd3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555743cc080 0x5555743d5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.547+0000 7f7decbd3740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555743d11f0 0x5555743dcfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.547+0000 7f7decbd3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5555743d4140 con 0x5555743cb690 2024-04-26T21:34:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.547+0000 7f7decbd3740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5555742b7fe0 con 0x5555743d11f0 2024-04-26T21:34:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.547+0000 7f7decbd3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5555743d42c0 con 0x5555743cc080 2024-04-26T21:34:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.547+0000 7f7decbd3740 10 client.0 _ll_register_callbacks cb 0x5555742ce150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:34:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.547+0000 7f7de37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743cb690 0x5555743d5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.547+0000 7f7de37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743cb690 0x5555743d5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40490/0 (socket says 172.21.15.46:40490) 2024-04-26T21:34:15.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.547+0000 7f7de37fe640 1 -- 172.21.15.46:0/2607901271 learned_addr learned my addr 172.21.15.46:0/2607901271 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:15.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.547+0000 7f7de2ffd640 1 --2- 172.21.15.46:0/2607901271 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555743cc080 0x5555743d5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.547+0000 7f7de3fff640 1 --2- 172.21.15.46:0/2607901271 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555743d11f0 0x5555743dcfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.547+0000 7f7de37fe640 1 -- 172.21.15.46:0/2607901271 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555743cc080 msgr2=0x5555743d5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:15.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.547+0000 7f7de37fe640 1 --2- 172.21.15.46:0/2607901271 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555743cc080 0x5555743d5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.547+0000 7f7de37fe640 1 -- 172.21.15.46:0/2607901271 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555743d11f0 msgr2=0x5555743dcfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:15.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.547+0000 7f7de37fe640 1 --2- 172.21.15.46:0/2607901271 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555743d11f0 0x5555743dcfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.547+0000 7f7de37fe640 1 -- 172.21.15.46:0/2607901271 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555740422d0 con 0x5555743cb690 2024-04-26T21:34:15.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.547+0000 7f7de37fe640 1 --2- 172.21.15.46:0/2607901271 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743cb690 0x5555743d5a50 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4009610 tx=0x7f7dd406bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f33577caaee7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:15.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.547+0000 7f7dc2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-26T21:34:15.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.547+0000 7f7dc2ffd640 1 -- 172.21.15.46:0/2607901271 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4020070 con 0x5555743cb690 2024-04-26T21:34:15.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.547+0000 7f7dc2ffd640 1 -- 172.21.15.46:0/2607901271 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f7dd4098840 con 0x5555743cb690 2024-04-26T21:34:15.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.547+0000 7f7decbd3740 1 -- 172.21.15.46:0/2607901271 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7dcc040070 con 0x5555743cb690 2024-04-26T21:34:15.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.548+0000 7f7dc2ffd640 1 -- 172.21.15.46:0/2607901271 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f7dd40a0580 con 0x5555743cb690 2024-04-26T21:34:15.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.548+0000 7f7decbd3740 10 client.15114 fetch_fsmap learned FSMap version 22 2024-04-26T21:34:15.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.548+0000 7f7decbd3740 1 -- 172.21.15.46:0/2607901271 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x55557410c6b0 con 0x5555743cb690 2024-04-26T21:34:15.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.548+0000 7f7dc2ffd640 1 -- 172.21.15.46:0/2607901271 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7dcc040070 con 0x5555743cb690 2024-04-26T21:34:15.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.548+0000 7f7dc2ffd640 1 -- 172.21.15.46:0/2607901271 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f7dd40d2140 con 0x5555743cb690 2024-04-26T21:34:15.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.548+0000 7f7decbd3740 10 client.15114 fetch_fsmap finished waiting for FSMap version 22 2024-04-26T21:34:15.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.548+0000 7f7decbd3740 10 client.15114 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T21:34:15.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.548+0000 7f7decbd3740 20 client.15114 populate_metadata read hostname 'smithi046' 2024-04-26T21:34:15.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.548+0000 7f7decbd3740 1 --2- 172.21.15.46:0/2607901271 >> [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] conn(0x5555743e24a0 0x5555743e3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.548+0000 7f7decbd3740 4 client.15114 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:34:15.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.548+0000 7f7decbd3740 1 -- 172.21.15.46:0/2607901271 --> [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55557433e350 con 0x5555743e24a0 2024-04-26T21:34:15.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.548+0000 7f7de2ffd640 1 --2- 172.21.15.46:0/2607901271 >> [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] conn(0x5555743e24a0 0x5555743e3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.549+0000 7f7de2ffd640 1 --2- 172.21.15.46:0/2607901271 >> [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] conn(0x5555743e24a0 0x5555743e3910 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c30e3bb0aaad336 server_cookie=4f64b72efcdb589f in_seq=0 out_seq=0 2024-04-26T21:34:15.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.549+0000 7f7dc2ffd640 10 client.15114 ms_handle_connect on v2:172.21.15.50:6838/2453253368 2024-04-26T21:34:15.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.550+0000 7f7dc2ffd640 1 -- 172.21.15.46:0/2607901271 <== mds.0 v2:172.21.15.50:6838/2453253368 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55557433e350 con 0x5555743e24a0 2024-04-26T21:34:15.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.550+0000 7f7dc2ffd640 10 client.15114 handle_command_reply: tid=0 2024-04-26T21:34:15.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.550+0000 7f7decbd3740 1 client.15114 shutdown 2024-04-26T21:34:15.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.550+0000 7f7decbd3740 1 -- 172.21.15.46:0/2607901271 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743cb690 msgr2=0x5555743d5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.550+0000 7f7decbd3740 1 --2- 172.21.15.46:0/2607901271 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743cb690 0x5555743d5a50 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4009610 tx=0x7f7dd406bb00 comp rx=0 tx=0).stop 2024-04-26T21:34:15.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.550+0000 7f7decbd3740 1 -- 172.21.15.46:0/2607901271 shutdown_connections 2024-04-26T21:34:15.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.550+0000 7f7decbd3740 1 --2- 172.21.15.46:0/2607901271 >> [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] conn(0x5555743e24a0 0x5555743e3910 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.550+0000 7f7decbd3740 1 --2- 172.21.15.46:0/2607901271 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555743d11f0 0x5555743dcfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.550+0000 7f7decbd3740 1 --2- 172.21.15.46:0/2607901271 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555743cc080 0x5555743d5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.550+0000 7f7decbd3740 1 --2- 172.21.15.46:0/2607901271 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743cb690 0x5555743d5a50 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.550+0000 7f7dc2ffd640 0 client.15114 ms_handle_reset on v2:172.21.15.50:6838/2453253368 2024-04-26T21:34:15.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.550+0000 7f7decbd3740 1 -- 172.21.15.46:0/2607901271 >> 172.21.15.46:0/2607901271 conn(0x5555743c6d00 msgr2=0x5555743c5f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:15.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.550+0000 7f7decbd3740 1 -- 172.21.15.46:0/2607901271 shutdown_connections 2024-04-26T21:34:15.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.550+0000 7f7decbd3740 1 -- 172.21.15.46:0/2607901271 wait complete. 2024-04-26T21:34:15.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.550+0000 7f7decbd3740 20 client.15114 trim_cache size 0 max 16384 2024-04-26T21:34:15.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.568+0000 7f7decbd3740 1 Processor -- start 2024-04-26T21:34:15.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.568+0000 7f7decbd3740 1 -- start start 2024-04-26T21:34:15.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.568+0000 7f7decbd3740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555743cb690 0x5555743d5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.568+0000 7f7decbd3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555743cc080 0x5555743dcf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.568+0000 7f7decbd3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743d11f0 0x5555743d5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.568+0000 7f7decbd3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5555743e1c50 con 0x5555743d11f0 2024-04-26T21:34:15.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.568+0000 7f7decbd3740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5555743e2320 con 0x5555743cb690 2024-04-26T21:34:15.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.568+0000 7f7decbd3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5555743e29f0 con 0x5555743cc080 2024-04-26T21:34:15.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.569+0000 7f7de3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743d11f0 0x5555743d5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.569+0000 7f7de3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743d11f0 0x5555743d5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40498/0 (socket says 172.21.15.46:40498) 2024-04-26T21:34:15.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.569+0000 7f7de3fff640 1 -- 172.21.15.46:0/3469433824 learned_addr learned my addr 172.21.15.46:0/3469433824 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:15.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.569+0000 7f7de37fe640 1 --2- 172.21.15.46:0/3469433824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555743cb690 0x5555743d5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.569+0000 7f7de3fff640 1 -- 172.21.15.46:0/3469433824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555743cc080 msgr2=0x5555743dcf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:15.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.569+0000 7f7de3fff640 1 --2- 172.21.15.46:0/3469433824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555743cc080 0x5555743dcf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.569+0000 7f7de3fff640 1 -- 172.21.15.46:0/3469433824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555743cb690 msgr2=0x5555743d5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:15.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.569+0000 7f7de3fff640 1 --2- 172.21.15.46:0/3469433824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555743cb690 0x5555743d5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.569+0000 7f7de3fff640 1 -- 172.21.15.46:0/3469433824 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555743d0750 con 0x5555743d11f0 2024-04-26T21:34:15.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.569+0000 7f7de37fe640 1 --2- 172.21.15.46:0/3469433824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555743cb690 0x5555743d5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:34:15.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.569+0000 7f7de3fff640 1 --2- 172.21.15.46:0/3469433824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743d11f0 0x5555743d5ba0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc011a00 tx=0x7f7ddc009890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9985f5f86d99ac5a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:15.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.569+0000 7f7dd97fa640 1 -- 172.21.15.46:0/3469433824 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc019070 con 0x5555743d11f0 2024-04-26T21:34:15.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.569+0000 7f7dd97fa640 1 -- 172.21.15.46:0/3469433824 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ddc021c50 con 0x5555743d11f0 2024-04-26T21:34:15.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.569+0000 7f7dd97fa640 1 -- 172.21.15.46:0/3469433824 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc010ea0 con 0x5555743d11f0 2024-04-26T21:34:15.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.570+0000 7f7decbd3740 1 -- 172.21.15.46:0/3469433824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743d11f0 msgr2=0x5555743d5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.570+0000 7f7decbd3740 1 --2- 172.21.15.46:0/3469433824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743d11f0 0x5555743d5ba0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc011a00 tx=0x7f7ddc009890 comp rx=0 tx=0).stop 2024-04-26T21:34:15.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.570+0000 7f7decbd3740 1 -- 172.21.15.46:0/3469433824 shutdown_connections 2024-04-26T21:34:15.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.570+0000 7f7decbd3740 1 --2- 172.21.15.46:0/3469433824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743d11f0 0x5555743d5ba0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.570+0000 7f7decbd3740 1 --2- 172.21.15.46:0/3469433824 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555743cc080 0x5555743dcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.570+0000 7f7decbd3740 1 --2- 172.21.15.46:0/3469433824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555743cb690 0x5555743d5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.570+0000 7f7decbd3740 1 -- 172.21.15.46:0/3469433824 >> 172.21.15.46:0/3469433824 conn(0x5555743c6d00 msgr2=0x5555743d4660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:15.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.570+0000 7f7decbd3740 1 -- 172.21.15.46:0/3469433824 shutdown_connections 2024-04-26T21:34:15.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.570+0000 7f7decbd3740 1 -- 172.21.15.46:0/3469433824 wait complete. 2024-04-26T21:34:15.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.570+0000 7f7decbd3740 1 -- start start 2024-04-26T21:34:15.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.570+0000 7f7decbd3740 1 Processor -- start 2024-04-26T21:34:15.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.570+0000 7f7decbd3740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555743cb690 0x5555743d9b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.570+0000 7f7decbd3740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555743cc080 0x5555743da090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.570+0000 7f7decbd3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743d11f0 0x555574448450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.570+0000 7f7decbd3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5555743e3790 con 0x5555743d11f0 2024-04-26T21:34:15.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.570+0000 7f7decbd3740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5555743e30c0 con 0x5555743cc080 2024-04-26T21:34:15.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.570+0000 7f7decbd3740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5555743e3e60 con 0x5555743cb690 2024-04-26T21:34:15.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.571+0000 7f7decbd3740 10 client.0 _ll_register_callbacks cb 0x7f7dd400d6c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:34:15.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.571+0000 7f7de3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743d11f0 0x555574448450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.571+0000 7f7de3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743d11f0 0x555574448450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40504/0 (socket says 172.21.15.46:40504) 2024-04-26T21:34:15.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.571+0000 7f7de3fff640 1 -- 172.21.15.46:0/3436434908 learned_addr learned my addr 172.21.15.46:0/3436434908 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:15.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.571+0000 7f7de37fe640 1 --2- 172.21.15.46:0/3436434908 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555743cb690 0x5555743d9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.571+0000 7f7de2ffd640 1 --2- 172.21.15.46:0/3436434908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555743cc080 0x5555743da090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.571+0000 7f7de3fff640 1 -- 172.21.15.46:0/3436434908 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555743cb690 msgr2=0x5555743d9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:15.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.571+0000 7f7de3fff640 1 --2- 172.21.15.46:0/3436434908 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555743cb690 0x5555743d9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.571+0000 7f7de3fff640 1 -- 172.21.15.46:0/3436434908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555743cc080 msgr2=0x5555743da090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:15.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.571+0000 7f7de3fff640 1 --2- 172.21.15.46:0/3436434908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555743cc080 0x5555743da090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.571+0000 7f7de3fff640 1 -- 172.21.15.46:0/3436434908 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555743dd160 con 0x5555743d11f0 2024-04-26T21:34:15.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.571+0000 7f7de37fe640 1 --2- 172.21.15.46:0/3436434908 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555743cb690 0x5555743d9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:34:15.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.571+0000 7f7de3fff640 1 --2- 172.21.15.46:0/3436434908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743d11f0 0x555574448450 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc0119d0 tx=0x7f7ddc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79a64b5b42a35ffa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:15.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.571+0000 7f7dc2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-26T21:34:15.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.571+0000 7f7dc2ffd640 1 -- 172.21.15.46:0/3436434908 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc019040 con 0x5555743d11f0 2024-04-26T21:34:15.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.571+0000 7f7dc2ffd640 1 -- 172.21.15.46:0/3436434908 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f7ddc021e00 con 0x5555743d11f0 2024-04-26T21:34:15.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.571+0000 7f7decbd3740 1 -- 172.21.15.46:0/3436434908 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ddc021c50 con 0x5555743d11f0 2024-04-26T21:34:15.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.571+0000 7f7dc2ffd640 1 -- 172.21.15.46:0/3436434908 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f7ddc010660 con 0x5555743d11f0 2024-04-26T21:34:15.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.571+0000 7f7decbd3740 10 client.15120 fetch_fsmap learned FSMap version 22 2024-04-26T21:34:15.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.571+0000 7f7decbd3740 1 -- 172.21.15.46:0/3436434908 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5555744497b0 con 0x5555743d11f0 2024-04-26T21:34:15.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.572+0000 7f7dc2ffd640 1 -- 172.21.15.46:0/3436434908 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ddc021c50 con 0x5555743d11f0 2024-04-26T21:34:15.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.572+0000 7f7dc2ffd640 1 -- 172.21.15.46:0/3436434908 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f7ddc033070 con 0x5555743d11f0 2024-04-26T21:34:15.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.572+0000 7f7decbd3740 10 client.15120 fetch_fsmap finished waiting for FSMap version 22 2024-04-26T21:34:15.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.572+0000 7f7decbd3740 10 client.15120 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T21:34:15.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.572+0000 7f7decbd3740 20 client.15120 populate_metadata read hostname 'smithi046' 2024-04-26T21:34:15.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.572+0000 7f7decbd3740 1 --2- 172.21.15.46:0/3436434908 >> [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] conn(0x55557444e200 0x55557444cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:15.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.572+0000 7f7decbd3740 4 client.15120 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T21:34:15.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.572+0000 7f7decbd3740 1 -- 172.21.15.46:0/3436434908 --> [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] -- command(tid 0: {"prefix": "status"}) -- 0x5555743381c0 con 0x55557444e200 2024-04-26T21:34:15.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.572+0000 7f7de37fe640 1 --2- 172.21.15.46:0/3436434908 >> [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] conn(0x55557444e200 0x55557444cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:15.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.572+0000 7f7de37fe640 1 --2- 172.21.15.46:0/3436434908 >> [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] conn(0x55557444e200 0x55557444cf10 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34cc9f06788704d6 server_cookie=778ab216873e580 in_seq=0 out_seq=0 2024-04-26T21:34:15.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.572+0000 7f7dc2ffd640 10 client.15120 ms_handle_connect on v2:172.21.15.50:6838/2453253368 2024-04-26T21:34:15.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.573+0000 7f7dc2ffd640 1 -- 172.21.15.46:0/3436434908 <== mds.0 v2:172.21.15.50:6838/2453253368 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5555743381c0 con 0x55557444e200 2024-04-26T21:34:15.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.573+0000 7f7dc2ffd640 10 client.15120 handle_command_reply: tid=0 2024-04-26T21:34:15.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.573+0000 7f7decbd3740 1 client.15120 shutdown 2024-04-26T21:34:15.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.573+0000 7f7decbd3740 1 -- 172.21.15.46:0/3436434908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743d11f0 msgr2=0x555574448450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.573+0000 7f7decbd3740 1 --2- 172.21.15.46:0/3436434908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743d11f0 0x555574448450 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc0119d0 tx=0x7f7ddc004a40 comp rx=0 tx=0).stop 2024-04-26T21:34:15.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.573+0000 7f7decbd3740 1 -- 172.21.15.46:0/3436434908 shutdown_connections 2024-04-26T21:34:15.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.573+0000 7f7decbd3740 1 --2- 172.21.15.46:0/3436434908 >> [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] conn(0x55557444e200 0x55557444cf10 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.573+0000 7f7decbd3740 1 --2- 172.21.15.46:0/3436434908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5555743d11f0 0x555574448450 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.573+0000 7f7decbd3740 1 --2- 172.21.15.46:0/3436434908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5555743cc080 0x5555743da090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.573+0000 7f7dc2ffd640 0 client.15120 ms_handle_reset on v2:172.21.15.50:6838/2453253368 2024-04-26T21:34:15.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.573+0000 7f7decbd3740 1 --2- 172.21.15.46:0/3436434908 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5555743cb690 0x5555743d9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.573+0000 7f7decbd3740 1 -- 172.21.15.46:0/3436434908 >> 172.21.15.46:0/3436434908 conn(0x5555743c6d00 msgr2=0x5555743d5660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:15.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.573+0000 7f7decbd3740 1 -- 172.21.15.46:0/3436434908 shutdown_connections 2024-04-26T21:34:15.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.573+0000 7f7decbd3740 1 -- 172.21.15.46:0/3436434908 wait complete. 2024-04-26T21:34:15.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.573+0000 7f7decbd3740 20 client.15120 trim_cache size 0 max 16384 2024-04-26T21:34:15.582 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-26T21:34:15.582 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36c7013a-0413-11ef-bc93-c7b262605968", 2024-04-26T21:34:15.582 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-04-26T21:34:15.582 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 24539, 2024-04-26T21:34:15.582 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-04-26T21:34:15.582 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-04-26T21:34:15.582 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-04-26T21:34:15.582 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 90.614273971000003, 2024-04-26T21:34:15.582 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 22, 2024-04-26T21:34:15.583 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 87, 2024-04-26T21:34:15.583 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 85, 2024-04-26T21:34:15.583 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 96.500470664999995 2024-04-26T21:34:15.583 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-26T21:34:15.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.574+0000 7f7de9de4640 1 -- 172.21.15.46:0/2334886896 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7dd00a3c40 msgr2=0x7f7dd00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.574+0000 7f7de9de4640 1 --2- 172.21.15.46:0/2334886896 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7dd00a3c40 0x7f7dd00a6100 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x5555743cc8e0 tx=0x7f7ddc004ab0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.574+0000 7f7de9de4640 1 -- 172.21.15.46:0/2334886896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de4103410 msgr2=0x7f7de4102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:15.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.574+0000 7f7de9de4640 1 --2- 172.21.15.46:0/2334886896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de4103410 0x7f7de4102d30 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7dd400d770 tx=0x7f7dd400dc30 comp rx=0 tx=0).stop 2024-04-26T21:34:15.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.574+0000 7f7de9de4640 1 -- 172.21.15.46:0/2334886896 shutdown_connections 2024-04-26T21:34:15.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.574+0000 7f7de9de4640 1 --2- 172.21.15.46:0/2334886896 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f7dd00a3c40 0x7f7dd00a6100 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.574+0000 7f7de9de4640 1 --2- 172.21.15.46:0/2334886896 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7de4108670 0x7f7de4101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.574+0000 7f7de9de4640 1 --2- 172.21.15.46:0/2334886896 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7de4105d40 0x7f7de41013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.574+0000 7f7de9de4640 1 --2- 172.21.15.46:0/2334886896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7de4103410 0x7f7de4102d30 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:15.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.574+0000 7f7de9de4640 1 -- 172.21.15.46:0/2334886896 >> 172.21.15.46:0/2334886896 conn(0x7f7de40fce20 msgr2=0x7f7de4103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:15.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.574+0000 7f7de9de4640 1 -- 172.21.15.46:0/2334886896 shutdown_connections 2024-04-26T21:34:15.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:15.574+0000 7f7de9de4640 1 -- 172.21.15.46:0/2334886896 wait complete. 2024-04-26T21:34:15.586 INFO:tasks.cephadm:Setup complete, yielding 2024-04-26T21:34:15.586 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-26T21:34:15.596 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi046.front.sepia.ceph.com 2024-04-26T21:34:15.596 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-26T21:34:15.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:15 smithi046 ceph-mon[29201]: pgmap v339: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:34:15.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:15 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/770590354' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:34:15.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:15 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1257531182' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:34:15.842 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:34:15.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:15 smithi050 ceph-mon[32753]: pgmap v339: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:34:15.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:15 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/770590354' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:34:15.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:15 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/1257531182' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:34:16.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:15 smithi142 ceph-mon[32944]: pgmap v339: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:34:16.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:15 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/770590354' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:34:16.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:15 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/1257531182' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:34:17.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.727+0000 7f657253b700 1 -- 172.21.15.46:0/1410552734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f656c101750 msgr2=0x7f656c101b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.727+0000 7f657253b700 1 --2- 172.21.15.46:0/1410552734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f656c101750 0x7f656c101b70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f65540099c0 tx=0x7f6554009cd0 comp rx=0 tx=0).stop 2024-04-26T21:34:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.727+0000 7f657253b700 1 -- 172.21.15.46:0/1410552734 shutdown_connections 2024-04-26T21:34:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.727+0000 7f657253b700 1 --2- 172.21.15.46:0/1410552734 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f656c101750 0x7f656c101b70 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.727+0000 7f657253b700 1 --2- 172.21.15.46:0/1410552734 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f656c105170 0x7f656c10ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.727+0000 7f657253b700 1 --2- 172.21.15.46:0/1410552734 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f656c1047d0 0x7f656c104c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.727+0000 7f657253b700 1 -- 172.21.15.46:0/1410552734 >> 172.21.15.46:0/1410552734 conn(0x7f656c0fd0b0 msgr2=0x7f656c0ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.727+0000 7f657253b700 1 -- 172.21.15.46:0/1410552734 shutdown_connections 2024-04-26T21:34:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.727+0000 7f657253b700 1 -- 172.21.15.46:0/1410552734 wait complete. 2024-04-26T21:34:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.728+0000 7f657253b700 1 Processor -- start 2024-04-26T21:34:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.728+0000 7f657253b700 1 -- start start 2024-04-26T21:34:17.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.728+0000 7f657253b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f656c101750 0x7f656c199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.728+0000 7f657253b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f656c1047d0 0x7f656c199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.728+0000 7f657253b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f656c105170 0x7f656c1a0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.728+0000 7f657253b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f656c10ef00 con 0x7f656c1047d0 2024-04-26T21:34:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.728+0000 7f657253b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f656c10ed80 con 0x7f656c105170 2024-04-26T21:34:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.728+0000 7f657253b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f656c10f080 con 0x7f656c101750 2024-04-26T21:34:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.728+0000 7f656b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f656c1047d0 0x7f656c199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.728+0000 7f656b7fe700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f656c1047d0 0x7f656c199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59260/0 (socket says 172.21.15.46:59260) 2024-04-26T21:34:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.728+0000 7f656bfff700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f656c101750 0x7f656c199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.728+0000 7f656b7fe700 1 -- 172.21.15.46:0/3202116687 learned_addr learned my addr 172.21.15.46:0/3202116687 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.728+0000 7f6570ad8700 1 --2- 172.21.15.46:0/3202116687 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f656c105170 0x7f656c1a0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.729+0000 7f656b7fe700 1 -- 172.21.15.46:0/3202116687 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f656c101750 msgr2=0x7f656c199550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.729+0000 7f656b7fe700 1 --2- 172.21.15.46:0/3202116687 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f656c101750 0x7f656c199550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.729+0000 7f656b7fe700 1 -- 172.21.15.46:0/3202116687 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f656c105170 msgr2=0x7f656c1a0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.729+0000 7f656b7fe700 1 --2- 172.21.15.46:0/3202116687 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f656c105170 0x7f656c1a0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:17.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.729+0000 7f656b7fe700 1 -- 172.21.15.46:0/3202116687 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f656c1a11b0 con 0x7f656c1047d0 2024-04-26T21:34:17.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.729+0000 7f656bfff700 1 --2- 172.21.15.46:0/3202116687 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f656c101750 0x7f656c199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:34:17.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.729+0000 7f656b7fe700 1 --2- 172.21.15.46:0/3202116687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f656c1047d0 0x7f656c199a90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f655c00d980 tx=0x7f655c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.729+0000 7f65697fa700 1 -- 172.21.15.46:0/3202116687 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655c016070 con 0x7f656c1047d0 2024-04-26T21:34:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.729+0000 7f657253b700 1 -- 172.21.15.46:0/3202116687 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f656c1a14a0 con 0x7f656c1047d0 2024-04-26T21:34:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.729+0000 7f65697fa700 1 -- 172.21.15.46:0/3202116687 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f655c009d80 con 0x7f656c1047d0 2024-04-26T21:34:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.729+0000 7f657253b700 1 -- 172.21.15.46:0/3202116687 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f656c1a1a30 con 0x7f656c1047d0 2024-04-26T21:34:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.729+0000 7f65697fa700 1 -- 172.21.15.46:0/3202116687 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655c00f3c0 con 0x7f656c1047d0 2024-04-26T21:34:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.730+0000 7f657253b700 1 -- 172.21.15.46:0/3202116687 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f654c005360 con 0x7f656c1047d0 2024-04-26T21:34:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.731+0000 7f65697fa700 1 -- 172.21.15.46:0/3202116687 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f655c019070 con 0x7f656c1047d0 2024-04-26T21:34:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.732+0000 7f65697fa700 1 --2- 172.21.15.46:0/3202116687 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f65580a3d50 0x7f65580a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:17.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.732+0000 7f656bfff700 1 --2- 172.21.15.46:0/3202116687 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f65580a3d50 0x7f65580a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:17.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.732+0000 7f65697fa700 1 -- 172.21.15.46:0/3202116687 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f655c0d0240 con 0x7f656c1047d0 2024-04-26T21:34:17.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.732+0000 7f656bfff700 1 --2- 172.21.15.46:0/3202116687 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f65580a3d50 0x7f65580a6210 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6554006010 tx=0x7f655400bc20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:17.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.736+0000 7f65697fa700 1 -- 172.21.15.46:0/3202116687 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f655c09ca90 con 0x7f656c1047d0 2024-04-26T21:34:17.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:17 smithi050 ceph-mon[32753]: pgmap v340: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:34:17.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.962+0000 7f657253b700 1 -- 172.21.15.46:0/3202116687 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f654c000bf0 con 0x7f65580a3d50 2024-04-26T21:34:17.963 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:17 smithi046 ceph-mon[29201]: pgmap v340: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:34:17.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.963+0000 7f65697fa700 1 -- 172.21.15.46:0/3202116687 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f654c000bf0 con 0x7f65580a3d50 2024-04-26T21:34:17.963 INFO:teuthology.orchestra.run.smithi046.stdout:Backend: cephadm 2024-04-26T21:34:17.964 INFO:teuthology.orchestra.run.smithi046.stdout:Available: Yes 2024-04-26T21:34:17.964 INFO:teuthology.orchestra.run.smithi046.stdout:Paused: No 2024-04-26T21:34:17.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.965+0000 7f657253b700 1 -- 172.21.15.46:0/3202116687 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f65580a3d50 msgr2=0x7f65580a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:17.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.965+0000 7f657253b700 1 --2- 172.21.15.46:0/3202116687 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f65580a3d50 0x7f65580a6210 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6554006010 tx=0x7f655400bc20 comp rx=0 tx=0).stop 2024-04-26T21:34:17.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.965+0000 7f657253b700 1 -- 172.21.15.46:0/3202116687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f656c1047d0 msgr2=0x7f656c199a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:17.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.965+0000 7f657253b700 1 --2- 172.21.15.46:0/3202116687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f656c1047d0 0x7f656c199a90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f655c00d980 tx=0x7f655c00dc90 comp rx=0 tx=0).stop 2024-04-26T21:34:17.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.965+0000 7f657253b700 1 -- 172.21.15.46:0/3202116687 shutdown_connections 2024-04-26T21:34:17.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.965+0000 7f657253b700 1 --2- 172.21.15.46:0/3202116687 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f656c105170 0x7f656c1a0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:17.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.965+0000 7f657253b700 1 --2- 172.21.15.46:0/3202116687 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f656c101750 0x7f656c199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:17.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.965+0000 7f657253b700 1 --2- 172.21.15.46:0/3202116687 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f65580a3d50 0x7f65580a6210 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:17.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.965+0000 7f657253b700 1 --2- 172.21.15.46:0/3202116687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f656c1047d0 0x7f656c199a90 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:17.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.965+0000 7f657253b700 1 -- 172.21.15.46:0/3202116687 >> 172.21.15.46:0/3202116687 conn(0x7f656c0fd0b0 msgr2=0x7f656c107b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:17.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.965+0000 7f657253b700 1 -- 172.21.15.46:0/3202116687 shutdown_connections 2024-04-26T21:34:17.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:17.965+0000 7f657253b700 1 -- 172.21.15.46:0/3202116687 wait complete. 2024-04-26T21:34:18.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:17 smithi142 ceph-mon[32944]: pgmap v340: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:34:18.799 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-26T21:34:19.054 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:34:19.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:19 smithi046 ceph-mon[29201]: pgmap v341: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:19.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:19 smithi046 ceph-mon[29201]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:34:19.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:19 smithi050 ceph-mon[32753]: pgmap v341: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:19.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:19 smithi050 ceph-mon[32753]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:34:20.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:19 smithi142 ceph-mon[32944]: pgmap v341: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:20.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:19 smithi142 ceph-mon[32944]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:34:20.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.780+0000 7f83d54c2700 1 -- 172.21.15.46:0/1900289113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d0077450 msgr2=0x7f83d00758b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:20.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.780+0000 7f83d54c2700 1 --2- 172.21.15.46:0/1900289113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d0077450 0x7f83d00758b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f83c0009990 tx=0x7f83c0009ca0 comp rx=0 tx=0).stop 2024-04-26T21:34:20.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.780+0000 7f83d54c2700 1 -- 172.21.15.46:0/1900289113 shutdown_connections 2024-04-26T21:34:20.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.780+0000 7f83d54c2700 1 --2- 172.21.15.46:0/1900289113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d0077450 0x7f83d00758b0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:20.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.780+0000 7f83d54c2700 1 --2- 172.21.15.46:0/1900289113 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83d010fa50 0x7f83d0111e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:20.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.780+0000 7f83d54c2700 1 --2- 172.21.15.46:0/1900289113 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83d0075e80 0x7f83d0076300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:20.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.780+0000 7f83d54c2700 1 -- 172.21.15.46:0/1900289113 >> 172.21.15.46:0/1900289113 conn(0x7f83d00fea10 msgr2=0x7f83d0100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:20.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.780+0000 7f83d54c2700 1 -- 172.21.15.46:0/1900289113 shutdown_connections 2024-04-26T21:34:20.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.780+0000 7f83d54c2700 1 -- 172.21.15.46:0/1900289113 wait complete. 2024-04-26T21:34:20.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.781+0000 7f83d54c2700 1 Processor -- start 2024-04-26T21:34:20.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.781+0000 7f83d54c2700 1 -- start start 2024-04-26T21:34:20.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.781+0000 7f83d54c2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d0075e80 0x7f83d019fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:20.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.781+0000 7f83d54c2700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83d0077450 0x7f83d01a00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:20.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.781+0000 7f83d54c2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83d010fa50 0x7f83d01a7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:20.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.781+0000 7f83d54c2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f83d0115500 con 0x7f83d0075e80 2024-04-26T21:34:20.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.781+0000 7f83d54c2700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f83d0115380 con 0x7f83d010fa50 2024-04-26T21:34:20.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.781+0000 7f83d54c2700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f83d0115680 con 0x7f83d0077450 2024-04-26T21:34:20.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.781+0000 7f83ceffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d0075e80 0x7f83d019fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:20.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.781+0000 7f83ceffd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d0075e80 0x7f83d019fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59268/0 (socket says 172.21.15.46:59268) 2024-04-26T21:34:20.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.781+0000 7f83ceffd700 1 -- 172.21.15.46:0/2058249207 learned_addr learned my addr 172.21.15.46:0/2058249207 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:20.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.781+0000 7f83cf7fe700 1 --2- 172.21.15.46:0/2058249207 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83d010fa50 0x7f83d01a7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:20.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.781+0000 7f83ce7fc700 1 --2- 172.21.15.46:0/2058249207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83d0077450 0x7f83d01a00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:20.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.782+0000 7f83ceffd700 1 -- 172.21.15.46:0/2058249207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83d0077450 msgr2=0x7f83d01a00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:20.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.782+0000 7f83ceffd700 1 --2- 172.21.15.46:0/2058249207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83d0077450 0x7f83d01a00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:20.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.782+0000 7f83ceffd700 1 -- 172.21.15.46:0/2058249207 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83d010fa50 msgr2=0x7f83d01a7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:20.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.782+0000 7f83ceffd700 1 --2- 172.21.15.46:0/2058249207 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83d010fa50 0x7f83d01a7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:20.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.782+0000 7f83ceffd700 1 -- 172.21.15.46:0/2058249207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d01a7800 con 0x7f83d0075e80 2024-04-26T21:34:20.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.782+0000 7f83ceffd700 1 --2- 172.21.15.46:0/2058249207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d0075e80 0x7f83d019fba0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f83c00047b0 tx=0x7f83c0005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:20.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.782+0000 7f83c7fff700 1 -- 172.21.15.46:0/2058249207 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c0027070 con 0x7f83d0075e80 2024-04-26T21:34:20.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.782+0000 7f83d54c2700 1 -- 172.21.15.46:0/2058249207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83d01a7a90 con 0x7f83d0075e80 2024-04-26T21:34:20.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.782+0000 7f83c7fff700 1 -- 172.21.15.46:0/2058249207 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83c0004000 con 0x7f83d0075e80 2024-04-26T21:34:20.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.782+0000 7f83c7fff700 1 -- 172.21.15.46:0/2058249207 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c0022d10 con 0x7f83d0075e80 2024-04-26T21:34:20.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.782+0000 7f83d54c2700 1 -- 172.21.15.46:0/2058249207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83d01a7ee0 con 0x7f83d0075e80 2024-04-26T21:34:20.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.783+0000 7f83d54c2700 1 -- 172.21.15.46:0/2058249207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83d006a110 con 0x7f83d0075e80 2024-04-26T21:34:20.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.785+0000 7f83c7fff700 1 -- 172.21.15.46:0/2058249207 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f83c002a070 con 0x7f83d0075e80 2024-04-26T21:34:20.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.785+0000 7f83c7fff700 1 --2- 172.21.15.46:0/2058249207 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f83bc0a3cd0 0x7f83bc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:20.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.785+0000 7f83ce7fc700 1 --2- 172.21.15.46:0/2058249207 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f83bc0a3cd0 0x7f83bc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:20.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.785+0000 7f83c7fff700 1 -- 172.21.15.46:0/2058249207 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f83c00d16d0 con 0x7f83d0075e80 2024-04-26T21:34:20.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.786+0000 7f83ce7fc700 1 --2- 172.21.15.46:0/2058249207 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f83bc0a3cd0 0x7f83bc0a6190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f83d0077180 tx=0x7f83b800a3b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:20.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:20.789+0000 7f83c7fff700 1 -- 172.21.15.46:0/2058249207 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f83c0013070 con 0x7f83d0075e80 2024-04-26T21:34:21.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:21.012+0000 7f83d54c2700 1 -- 172.21.15.46:0/2058249207 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f83d0063860 con 0x7f83bc0a3cd0 2024-04-26T21:34:21.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:21.026+0000 7f83c7fff700 1 -- 172.21.15.46:0/2058249207 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7f83d0063860 con 0x7f83bc0a3cd0 2024-04-26T21:34:21.027 INFO:teuthology.orchestra.run.smithi046.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-26T21:34:21.027 INFO:teuthology.orchestra.run.smithi046.stdout:mds.a smithi046 running (105s) 18s ago 104s 15.4M - 19.0.0-3306-gbc612ffb 55dc54019ad8 28ef6aefc492 2024-04-26T21:34:21.027 INFO:teuthology.orchestra.run.smithi046.stdout:mds.b smithi050 running (102s) 35s ago 101s 26.0M - 19.0.0-3306-gbc612ffb 55dc54019ad8 f0907047d44e 2024-04-26T21:34:21.027 INFO:teuthology.orchestra.run.smithi046.stdout:mds.c smithi142 running (117s) 35s ago 116s 20.1M - 19.0.0-3306-gbc612ffb 55dc54019ad8 365169d6c55a 2024-04-26T21:34:21.027 INFO:teuthology.orchestra.run.smithi046.stdout:mds.d smithi046 running (2m) 18s ago 2m 16.4M - 19.0.0-3306-gbc612ffb 55dc54019ad8 04bfdfb73727 2024-04-26T21:34:21.027 INFO:teuthology.orchestra.run.smithi046.stdout:mds.e smithi050 running (2m) 35s ago 2m 18.3M - 19.0.0-3306-gbc612ffb 55dc54019ad8 9ba3ee1dbc35 2024-04-26T21:34:21.027 INFO:teuthology.orchestra.run.smithi046.stdout:mds.f smithi142 running (119s) 35s ago 119s 16.9M - 19.0.0-3306-gbc612ffb 55dc54019ad8 2bf98b6eacb3 2024-04-26T21:34:21.027 INFO:teuthology.orchestra.run.smithi046.stdout:mds.g smithi046 running (108s) 18s ago 107s 12.8M - 19.0.0-3306-gbc612ffb 55dc54019ad8 d4aed68ced68 2024-04-26T21:34:21.027 INFO:teuthology.orchestra.run.smithi046.stdout:mds.h smithi050 running (114s) 35s ago 113s 16.7M - 19.0.0-3306-gbc612ffb 55dc54019ad8 84e5daa6aa33 2024-04-26T21:34:21.028 INFO:teuthology.orchestra.run.smithi046.stdout:mds.i smithi142 running (111s) 35s ago 110s 20.4M - 19.0.0-3306-gbc612ffb 55dc54019ad8 5e031c0f66ac 2024-04-26T21:34:21.028 INFO:teuthology.orchestra.run.smithi046.stdout:mds.j smithi046 running (2m) 18s ago 2m 16.5M - 19.0.0-3306-gbc612ffb 55dc54019ad8 a38069bfbd62 2024-04-26T21:34:21.028 INFO:teuthology.orchestra.run.smithi046.stdout:mds.k smithi050 running (2m) 35s ago 2m 14.1M - 19.0.0-3306-gbc612ffb 55dc54019ad8 07c58350ace4 2024-04-26T21:34:21.028 INFO:teuthology.orchestra.run.smithi046.stdout:mds.l smithi142 running (2m) 35s ago 2m 16.2M - 19.0.0-3306-gbc612ffb 55dc54019ad8 de319134d991 2024-04-26T21:34:21.028 INFO:teuthology.orchestra.run.smithi046.stdout:mgr.x smithi046 *:9283,8765 running (9m) 18s ago 9m 516M - 19.0.0-3306-gbc612ffb 55dc54019ad8 d91b1acc0a72 2024-04-26T21:34:21.028 INFO:teuthology.orchestra.run.smithi046.stdout:mgr.y smithi050 *:8443,8765 running (7m) 35s ago 7m 454M - 19.0.0-3306-gbc612ffb 55dc54019ad8 9a30793b0e45 2024-04-26T21:34:21.028 INFO:teuthology.orchestra.run.smithi046.stdout:mgr.z smithi142 *:8443,8765 running (7m) 35s ago 7m 454M - 19.0.0-3306-gbc612ffb 55dc54019ad8 7662cfb726a5 2024-04-26T21:34:21.028 INFO:teuthology.orchestra.run.smithi046.stdout:mon.a smithi046 running (9m) 18s ago 9m 50.4M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 fc596ea263bf 2024-04-26T21:34:21.028 INFO:teuthology.orchestra.run.smithi046.stdout:mon.b smithi050 running (7m) 35s ago 7m 39.7M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 7e5f775ded48 2024-04-26T21:34:21.028 INFO:teuthology.orchestra.run.smithi046.stdout:mon.c smithi142 running (8m) 35s ago 8m 40.2M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 633ca5d086a4 2024-04-26T21:34:21.028 INFO:teuthology.orchestra.run.smithi046.stdout:osd.0 smithi046 running (7m) 18s ago 7m 46.1M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 91c7c1816fbb 2024-04-26T21:34:21.029 INFO:teuthology.orchestra.run.smithi046.stdout:osd.1 smithi050 running (6m) 35s ago 6m 56.5M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 14a8d8b3fd1c 2024-04-26T21:34:21.029 INFO:teuthology.orchestra.run.smithi046.stdout:osd.2 smithi142 running (6m) 35s ago 6m 48.2M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 fe18ac9debe9 2024-04-26T21:34:21.029 INFO:teuthology.orchestra.run.smithi046.stdout:osd.3 smithi046 running (6m) 18s ago 6m 54.3M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 2b88489ed663 2024-04-26T21:34:21.029 INFO:teuthology.orchestra.run.smithi046.stdout:osd.4 smithi050 running (5m) 35s ago 5m 55.4M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 d3b2bedf1fb1 2024-04-26T21:34:21.029 INFO:teuthology.orchestra.run.smithi046.stdout:osd.5 smithi142 running (5m) 35s ago 5m 52.3M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 fb8f44a8a152 2024-04-26T21:34:21.029 INFO:teuthology.orchestra.run.smithi046.stdout:osd.6 smithi046 running (4m) 18s ago 4m 44.5M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 8d397069bb82 2024-04-26T21:34:21.029 INFO:teuthology.orchestra.run.smithi046.stdout:osd.7 smithi050 running (4m) 35s ago 4m 46.1M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 f6a60f9bc776 2024-04-26T21:34:21.029 INFO:teuthology.orchestra.run.smithi046.stdout:osd.8 smithi142 running (3m) 35s ago 3m 53.8M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 5363d0630b2e 2024-04-26T21:34:21.029 INFO:teuthology.orchestra.run.smithi046.stdout:osd.9 smithi046 running (3m) 18s ago 3m 41.6M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 9f43b5373b15 2024-04-26T21:34:21.029 INFO:teuthology.orchestra.run.smithi046.stdout:osd.10 smithi050 running (3m) 35s ago 3m 57.3M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 5ddb6d50850f 2024-04-26T21:34:21.029 INFO:teuthology.orchestra.run.smithi046.stdout:osd.11 smithi142 running (2m) 35s ago 2m 52.0M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 64d973f436ec 2024-04-26T21:34:21.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:21.028+0000 7f83d54c2700 1 -- 172.21.15.46:0/2058249207 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f83bc0a3cd0 msgr2=0x7f83bc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:21.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:21.028+0000 7f83d54c2700 1 --2- 172.21.15.46:0/2058249207 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f83bc0a3cd0 0x7f83bc0a6190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f83d0077180 tx=0x7f83b800a3b0 comp rx=0 tx=0).stop 2024-04-26T21:34:21.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:21.028+0000 7f83d54c2700 1 -- 172.21.15.46:0/2058249207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d0075e80 msgr2=0x7f83d019fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:21.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:21.028+0000 7f83d54c2700 1 --2- 172.21.15.46:0/2058249207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d0075e80 0x7f83d019fba0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f83c00047b0 tx=0x7f83c0005b00 comp rx=0 tx=0).stop 2024-04-26T21:34:21.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:21.028+0000 7f83d54c2700 1 -- 172.21.15.46:0/2058249207 shutdown_connections 2024-04-26T21:34:21.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:21.028+0000 7f83d54c2700 1 --2- 172.21.15.46:0/2058249207 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f83bc0a3cd0 0x7f83bc0a6190 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:21.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:21.028+0000 7f83d54c2700 1 --2- 172.21.15.46:0/2058249207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d0075e80 0x7f83d019fba0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:21.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:21.028+0000 7f83d54c2700 1 --2- 172.21.15.46:0/2058249207 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f83d010fa50 0x7f83d01a7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:21.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:21.028+0000 7f83d54c2700 1 --2- 172.21.15.46:0/2058249207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f83d0077450 0x7f83d01a00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:21.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:21.028+0000 7f83d54c2700 1 -- 172.21.15.46:0/2058249207 >> 172.21.15.46:0/2058249207 conn(0x7f83d00fea10 msgr2=0x7f83d0102df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:21.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:21.028+0000 7f83d54c2700 1 -- 172.21.15.46:0/2058249207 shutdown_connections 2024-04-26T21:34:21.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:21.028+0000 7f83d54c2700 1 -- 172.21.15.46:0/2058249207 wait complete. 2024-04-26T21:34:21.715 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:21 smithi046 ceph-mon[29201]: pgmap v342: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:34:21.745 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-26T21:34:21.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:21 smithi050 ceph-mon[32753]: pgmap v342: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:34:22.026 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:34:22.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:21 smithi142 ceph-mon[32944]: pgmap v342: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:34:22.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:22 smithi046 ceph-mon[29201]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:34:23.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:22 smithi142 ceph-mon[32944]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:34:23.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:22 smithi050 ceph-mon[32753]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:34:23.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.801+0000 7fa5f7e88700 1 -- 172.21.15.46:0/895228041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5f01052d0 msgr2=0x7fa5f0105730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:23.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.801+0000 7fa5f7e88700 1 --2- 172.21.15.46:0/895228041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5f01052d0 0x7fa5f0105730 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0009990 tx=0x7fa5e0009ca0 comp rx=0 tx=0).stop 2024-04-26T21:34:23.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.801+0000 7fa5f7e88700 1 -- 172.21.15.46:0/895228041 shutdown_connections 2024-04-26T21:34:23.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.801+0000 7fa5f7e88700 1 --2- 172.21.15.46:0/895228041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5f0105c70 0x7fa5f010db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:23.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.801+0000 7fa5f7e88700 1 --2- 172.21.15.46:0/895228041 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5f0106d90 0x7fa5f01071b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:23.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.801+0000 7fa5f7e88700 1 --2- 172.21.15.46:0/895228041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5f01052d0 0x7fa5f0105730 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:23.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.801+0000 7fa5f7e88700 1 -- 172.21.15.46:0/895228041 >> 172.21.15.46:0/895228041 conn(0x7fa5f0100c70 msgr2=0x7fa5f0103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:23.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.801+0000 7fa5f7e88700 1 -- 172.21.15.46:0/895228041 shutdown_connections 2024-04-26T21:34:23.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.801+0000 7fa5f7e88700 1 -- 172.21.15.46:0/895228041 wait complete. 2024-04-26T21:34:23.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.801+0000 7fa5f7e88700 1 Processor -- start 2024-04-26T21:34:23.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.801+0000 7fa5f7e88700 1 -- start start 2024-04-26T21:34:23.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.802+0000 7fa5f7e88700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5f01052d0 0x7fa5f019b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:23.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.802+0000 7fa5f7e88700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5f0105c70 0x7fa5f019be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:23.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.802+0000 7fa5f7e88700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5f0106d90 0x7fa5f01a2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:23.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.802+0000 7fa5f7e88700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f0111290 con 0x7fa5f0105c70 2024-04-26T21:34:23.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.802+0000 7fa5f7e88700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f0111110 con 0x7fa5f0106d90 2024-04-26T21:34:23.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.802+0000 7fa5f7e88700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f0111410 con 0x7fa5f01052d0 2024-04-26T21:34:23.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.802+0000 7fa5f5423700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5f0105c70 0x7fa5f019be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:23.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.802+0000 7fa5f5423700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5f0105c70 0x7fa5f019be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59290/0 (socket says 172.21.15.46:59290) 2024-04-26T21:34:23.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.802+0000 7fa5f5423700 1 -- 172.21.15.46:0/1556304872 learned_addr learned my addr 172.21.15.46:0/1556304872 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:23.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.802+0000 7fa5f6425700 1 --2- 172.21.15.46:0/1556304872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5f0106d90 0x7fa5f01a2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:23.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.802+0000 7fa5f5c24700 1 --2- 172.21.15.46:0/1556304872 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5f01052d0 0x7fa5f019b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:23.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.802+0000 7fa5f5423700 1 -- 172.21.15.46:0/1556304872 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5f01052d0 msgr2=0x7fa5f019b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:23.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.802+0000 7fa5f5423700 1 --2- 172.21.15.46:0/1556304872 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5f01052d0 0x7fa5f019b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.802+0000 7fa5f5423700 1 -- 172.21.15.46:0/1556304872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5f0106d90 msgr2=0x7fa5f01a2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.802+0000 7fa5f5423700 1 --2- 172.21.15.46:0/1556304872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5f0106d90 0x7fa5f01a2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.802+0000 7fa5f5423700 1 -- 172.21.15.46:0/1556304872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5f01a3570 con 0x7fa5f0105c70 2024-04-26T21:34:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.802+0000 7fa5f5c24700 1 --2- 172.21.15.46:0/1556304872 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5f01052d0 0x7fa5f019b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:34:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.802+0000 7fa5f6425700 1 --2- 172.21.15.46:0/1556304872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5f0106d90 0x7fa5f01a2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:34:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.802+0000 7fa5f5423700 1 --2- 172.21.15.46:0/1556304872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5f0105c70 0x7fa5f019be50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fa5e000b720 tx=0x7fa5e00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.802+0000 7fa5e6ffd700 1 -- 172.21.15.46:0/1556304872 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e0026070 con 0x7fa5f0105c70 2024-04-26T21:34:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.803+0000 7fa5e6ffd700 1 -- 172.21.15.46:0/1556304872 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5e000fc40 con 0x7fa5f0105c70 2024-04-26T21:34:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.803+0000 7fa5f7e88700 1 -- 172.21.15.46:0/1556304872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5f01a3800 con 0x7fa5f0105c70 2024-04-26T21:34:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.803+0000 7fa5e6ffd700 1 -- 172.21.15.46:0/1556304872 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e0017650 con 0x7fa5f0105c70 2024-04-26T21:34:23.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.803+0000 7fa5f7e88700 1 -- 172.21.15.46:0/1556304872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5f01a3c50 con 0x7fa5f0105c70 2024-04-26T21:34:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.803+0000 7fa5f7e88700 1 -- 172.21.15.46:0/1556304872 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5f006a110 con 0x7fa5f0105c70 2024-04-26T21:34:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.805+0000 7fa5e6ffd700 1 -- 172.21.15.46:0/1556304872 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa5e0029070 con 0x7fa5f0105c70 2024-04-26T21:34:23.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.805+0000 7fa5e6ffd700 1 --2- 172.21.15.46:0/1556304872 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fa5dc0a3cd0 0x7fa5dc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.806+0000 7fa5f5c24700 1 --2- 172.21.15.46:0/1556304872 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fa5dc0a3cd0 0x7fa5dc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.806+0000 7fa5e6ffd700 1 -- 172.21.15.46:0/1556304872 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7fa5e00d19a0 con 0x7fa5f0105c70 2024-04-26T21:34:23.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.806+0000 7fa5f5c24700 1 --2- 172.21.15.46:0/1556304872 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fa5dc0a3cd0 0x7fa5dc0a6190 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0106710 tx=0x7fa5ec009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:23.808 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:23 smithi046 ceph-mon[29201]: pgmap v343: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:23.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:23.809+0000 7fa5e6ffd700 1 -- 172.21.15.46:0/1556304872 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa5e0030070 con 0x7fa5f0105c70 2024-04-26T21:34:24.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:24.027+0000 7fa5f7e88700 1 -- 172.21.15.46:0/1556304872 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fa5f0063860 con 0x7fa5dc0a3cd0 2024-04-26T21:34:24.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:24.033+0000 7fa5e6ffd700 1 -- 172.21.15.46:0/1556304872 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fa5f0063860 con 0x7fa5dc0a3cd0 2024-04-26T21:34:24.034 INFO:teuthology.orchestra.run.smithi046.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-26T21:34:24.034 INFO:teuthology.orchestra.run.smithi046.stdout:mds.all 12/12 38s ago 2m smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi050=b;smithi050=e;smithi050=h;smithi050=k;smithi142=c;smithi142=f;smithi142=i;smithi142=l;count:12 2024-04-26T21:34:24.034 INFO:teuthology.orchestra.run.smithi046.stdout:mgr 3/3 38s ago 7m smithi046=x;smithi050=y;smithi142=z;count:3 2024-04-26T21:34:24.035 INFO:teuthology.orchestra.run.smithi046.stdout:mon 3/3 38s ago 8m smithi046:172.21.15.46=a;smithi050:172.21.15.50=b;smithi142:172.21.15.142=c;count:3 2024-04-26T21:34:24.035 INFO:teuthology.orchestra.run.smithi046.stdout:osd 12 38s ago - 2024-04-26T21:34:24.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:24.035+0000 7fa5f7e88700 1 -- 172.21.15.46:0/1556304872 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fa5dc0a3cd0 msgr2=0x7fa5dc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:24.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:24.035+0000 7fa5f7e88700 1 --2- 172.21.15.46:0/1556304872 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fa5dc0a3cd0 0x7fa5dc0a6190 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0106710 tx=0x7fa5ec009260 comp rx=0 tx=0).stop 2024-04-26T21:34:24.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:24.035+0000 7fa5f7e88700 1 -- 172.21.15.46:0/1556304872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5f0105c70 msgr2=0x7fa5f019be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:24.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:24.035+0000 7fa5f7e88700 1 --2- 172.21.15.46:0/1556304872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5f0105c70 0x7fa5f019be50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fa5e000b720 tx=0x7fa5e00047b0 comp rx=0 tx=0).stop 2024-04-26T21:34:24.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:24.035+0000 7fa5f7e88700 1 -- 172.21.15.46:0/1556304872 shutdown_connections 2024-04-26T21:34:24.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:24.035+0000 7fa5f7e88700 1 --2- 172.21.15.46:0/1556304872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5f0106d90 0x7fa5f01a2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:24.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:24.035+0000 7fa5f7e88700 1 --2- 172.21.15.46:0/1556304872 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fa5f01052d0 0x7fa5f019b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:24.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:24.035+0000 7fa5f7e88700 1 --2- 172.21.15.46:0/1556304872 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fa5dc0a3cd0 0x7fa5dc0a6190 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:24.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:24.035+0000 7fa5f7e88700 1 --2- 172.21.15.46:0/1556304872 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5f0105c70 0x7fa5f019be50 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:24.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:24.035+0000 7fa5f7e88700 1 -- 172.21.15.46:0/1556304872 >> 172.21.15.46:0/1556304872 conn(0x7fa5f0100c70 msgr2=0x7fa5f0109c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:24.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:24.035+0000 7fa5f7e88700 1 -- 172.21.15.46:0/1556304872 shutdown_connections 2024-04-26T21:34:24.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:24.035+0000 7fa5f7e88700 1 -- 172.21.15.46:0/1556304872 wait complete. 2024-04-26T21:34:24.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:23 smithi142 ceph-mon[32944]: pgmap v343: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:24.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:23 smithi050 ceph-mon[32753]: pgmap v343: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:24.811 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-26T21:34:25.061 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:34:25.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:25 smithi046 ceph-mon[29201]: pgmap v344: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:25.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:25 smithi046 ceph-mon[29201]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:34:26.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:25 smithi142 ceph-mon[32944]: pgmap v344: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:26.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:25 smithi142 ceph-mon[32944]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:34:26.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:25 smithi050 ceph-mon[32753]: pgmap v344: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:26.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:25 smithi050 ceph-mon[32753]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:34:26.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.910+0000 7f2bf3fbc700 1 -- 172.21.15.46:0/2427073774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bec1034a0 msgr2=0x7f2bec1038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:26.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.910+0000 7f2bf3fbc700 1 --2- 172.21.15.46:0/2427073774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bec1034a0 0x7f2bec1038c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2be80099c0 tx=0x7f2be8009cd0 comp rx=0 tx=0).stop 2024-04-26T21:34:26.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.910+0000 7f2bf3fbc700 1 -- 172.21.15.46:0/2427073774 shutdown_connections 2024-04-26T21:34:26.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.910+0000 7f2bf3fbc700 1 --2- 172.21.15.46:0/2427073774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bec1034a0 0x7f2bec1038c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:26.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.910+0000 7f2bf3fbc700 1 --2- 172.21.15.46:0/2427073774 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bec105040 0x7f2bec10b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:26.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.910+0000 7f2bf3fbc700 1 --2- 172.21.15.46:0/2427073774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2bec1046a0 0x7f2bec104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:26.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.910+0000 7f2bf3fbc700 1 -- 172.21.15.46:0/2427073774 >> 172.21.15.46:0/2427073774 conn(0x7f2bec0fea40 msgr2=0x7f2bec100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:26.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.910+0000 7f2bf3fbc700 1 -- 172.21.15.46:0/2427073774 shutdown_connections 2024-04-26T21:34:26.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.910+0000 7f2bf3fbc700 1 -- 172.21.15.46:0/2427073774 wait complete. 2024-04-26T21:34:26.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.911+0000 7f2bf3fbc700 1 Processor -- start 2024-04-26T21:34:26.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.911+0000 7f2bf3fbc700 1 -- start start 2024-04-26T21:34:26.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.911+0000 7f2bf3fbc700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2bec1034a0 0x7f2bec0771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:26.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.911+0000 7f2bf3fbc700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bec1046a0 0x7f2bec075810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:26.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.911+0000 7f2bf3fbc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bec105040 0x7f2bec075d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:26.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.911+0000 7f2bf3fbc700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f2bec10efa0 con 0x7f2bec1046a0 2024-04-26T21:34:26.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.911+0000 7f2bf3fbc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2bec10ee20 con 0x7f2bec105040 2024-04-26T21:34:26.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.911+0000 7f2bf3fbc700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f2bec10f120 con 0x7f2bec1034a0 2024-04-26T21:34:26.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.911+0000 7f2bf1557700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bec1046a0 0x7f2bec075810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:26.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.911+0000 7f2bf1557700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bec1046a0 0x7f2bec075810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48590/0 (socket says 172.21.15.46:48590) 2024-04-26T21:34:26.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.911+0000 7f2bf1557700 1 -- 172.21.15.46:0/941310896 learned_addr learned my addr 172.21.15.46:0/941310896 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:26.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.911+0000 7f2bf1d58700 1 --2- 172.21.15.46:0/941310896 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2bec1034a0 0x7f2bec0771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:26.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.911+0000 7f2bf2559700 1 --2- 172.21.15.46:0/941310896 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bec105040 0x7f2bec075d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:26.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.911+0000 7f2bf1557700 1 -- 172.21.15.46:0/941310896 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2bec1034a0 msgr2=0x7f2bec0771c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:26.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.911+0000 7f2bf1557700 1 --2- 172.21.15.46:0/941310896 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2bec1034a0 0x7f2bec0771c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:26.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.912+0000 7f2bf1557700 1 -- 172.21.15.46:0/941310896 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bec105040 msgr2=0x7f2bec075d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:26.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.912+0000 7f2bf1557700 1 --2- 172.21.15.46:0/941310896 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bec105040 0x7f2bec075d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:26.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.912+0000 7f2bf1557700 1 -- 172.21.15.46:0/941310896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bec076600 con 0x7f2bec1046a0 2024-04-26T21:34:26.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.912+0000 7f2bf1d58700 1 --2- 172.21.15.46:0/941310896 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2bec1034a0 0x7f2bec0771c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:34:26.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.912+0000 7f2bf2559700 1 --2- 172.21.15.46:0/941310896 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bec105040 0x7f2bec075d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:34:26.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.912+0000 7f2bf1557700 1 --2- 172.21.15.46:0/941310896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bec1046a0 0x7f2bec075810 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc00b530 tx=0x7f2bdc00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:26.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.912+0000 7f2be2ffd700 1 -- 172.21.15.46:0/941310896 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc015020 con 0x7f2bec1046a0 2024-04-26T21:34:26.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.912+0000 7f2be2ffd700 1 -- 172.21.15.46:0/941310896 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bdc011ce0 con 0x7f2bec1046a0 2024-04-26T21:34:26.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.912+0000 7f2bf3fbc700 1 -- 172.21.15.46:0/941310896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bec1a37e0 con 0x7f2bec1046a0 2024-04-26T21:34:26.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.912+0000 7f2be2ffd700 1 -- 172.21.15.46:0/941310896 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bdc010d20 con 0x7f2bec1046a0 2024-04-26T21:34:26.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.912+0000 7f2bf3fbc700 1 -- 172.21.15.46:0/941310896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bec1a3cf0 con 0x7f2bec1046a0 2024-04-26T21:34:26.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.913+0000 7f2bf3fbc700 1 -- 172.21.15.46:0/941310896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bec06a110 con 0x7f2bec1046a0 2024-04-26T21:34:26.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.914+0000 7f2be2ffd700 1 -- 172.21.15.46:0/941310896 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2bdc018020 con 0x7f2bec1046a0 2024-04-26T21:34:26.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.915+0000 7f2be2ffd700 1 --2- 172.21.15.46:0/941310896 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f2bd80a3d50 0x7f2bd80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:26.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.915+0000 7f2bf1d58700 1 --2- 172.21.15.46:0/941310896 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f2bd80a3d50 0x7f2bd80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:26.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.915+0000 7f2be2ffd700 1 -- 172.21.15.46:0/941310896 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f2bdc0d01a0 con 0x7f2bec1046a0 2024-04-26T21:34:26.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.915+0000 7f2bf1d58700 1 --2- 172.21.15.46:0/941310896 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f2bd80a3d50 0x7f2bd80a6210 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2be800b580 tx=0x7f2be8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:26.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:26.919+0000 7f2be2ffd700 1 -- 172.21.15.46:0/941310896 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2bdc01f020 con 0x7f2bec1046a0 2024-04-26T21:34:27.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:27.135+0000 7f2bf3fbc700 1 -- 172.21.15.46:0/941310896 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f2bec063860 con 0x7f2bd80a3d50 2024-04-26T21:34:27.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:27.137+0000 7f2be2ffd700 1 -- 172.21.15.46:0/941310896 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f2bec063860 con 0x7f2bd80a3d50 2024-04-26T21:34:27.137 INFO:teuthology.orchestra.run.smithi046.stdout:HOST ADDR LABELS STATUS 2024-04-26T21:34:27.137 INFO:teuthology.orchestra.run.smithi046.stdout:smithi046 172.21.15.46 2024-04-26T21:34:27.137 INFO:teuthology.orchestra.run.smithi046.stdout:smithi050 172.21.15.50 2024-04-26T21:34:27.138 INFO:teuthology.orchestra.run.smithi046.stdout:smithi142 172.21.15.142 2024-04-26T21:34:27.138 INFO:teuthology.orchestra.run.smithi046.stdout:3 hosts in cluster 2024-04-26T21:34:27.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:27.139+0000 7f2bf3fbc700 1 -- 172.21.15.46:0/941310896 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f2bd80a3d50 msgr2=0x7f2bd80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:27.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:27.139+0000 7f2bf3fbc700 1 --2- 172.21.15.46:0/941310896 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f2bd80a3d50 0x7f2bd80a6210 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2be800b580 tx=0x7f2be8005fb0 comp rx=0 tx=0).stop 2024-04-26T21:34:27.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:27.139+0000 7f2bf3fbc700 1 -- 172.21.15.46:0/941310896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bec1046a0 msgr2=0x7f2bec075810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:27.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:27.139+0000 7f2bf3fbc700 1 --2- 172.21.15.46:0/941310896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bec1046a0 0x7f2bec075810 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc00b530 tx=0x7f2bdc00b8f0 comp rx=0 tx=0).stop 2024-04-26T21:34:27.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:27.139+0000 7f2bf3fbc700 1 -- 172.21.15.46:0/941310896 shutdown_connections 2024-04-26T21:34:27.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:27.139+0000 7f2bf3fbc700 1 --2- 172.21.15.46:0/941310896 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bec105040 0x7f2bec075d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:27.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:27.139+0000 7f2bf3fbc700 1 --2- 172.21.15.46:0/941310896 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f2bec1034a0 0x7f2bec0771c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:27.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:27.139+0000 7f2bf3fbc700 1 --2- 172.21.15.46:0/941310896 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f2bd80a3d50 0x7f2bd80a6210 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:27.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:27.139+0000 7f2bf3fbc700 1 --2- 172.21.15.46:0/941310896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2bec1046a0 0x7f2bec075810 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:27.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:27.139+0000 7f2bf3fbc700 1 -- 172.21.15.46:0/941310896 >> 172.21.15.46:0/941310896 conn(0x7f2bec0fea40 msgr2=0x7f2bec1079c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:27.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:27.139+0000 7f2bf3fbc700 1 -- 172.21.15.46:0/941310896 shutdown_connections 2024-04-26T21:34:27.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:27.139+0000 7f2bf3fbc700 1 -- 172.21.15.46:0/941310896 wait complete. 2024-04-26T21:34:27.800 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:27 smithi046 ceph-mon[29201]: pgmap v345: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:34:27.834 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-26T21:34:28.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:27 smithi142 ceph-mon[32944]: pgmap v345: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:34:28.089 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:34:28.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:27 smithi050 ceph-mon[32753]: pgmap v345: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:34:28.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:28 smithi046 ceph-mon[29201]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:34:29.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:28 smithi142 ceph-mon[32944]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:34:29.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:28 smithi050 ceph-mon[32753]: from='client.15144 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:34:29.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:29 smithi046 ceph-mon[29201]: pgmap v346: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:30.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:29 smithi142 ceph-mon[32944]: pgmap v346: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:30.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.069+0000 7f936ff2b700 1 -- 172.21.15.46:0/2078712460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9368105020 msgr2=0x7f936810b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:30.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.069+0000 7f936ff2b700 1 --2- 172.21.15.46:0/2078712460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9368105020 0x7f936810b8d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f935800b0d0 tx=0x7f935800b3e0 comp rx=0 tx=0).stop 2024-04-26T21:34:30.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.069+0000 7f936ff2b700 1 -- 172.21.15.46:0/2078712460 shutdown_connections 2024-04-26T21:34:30.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.069+0000 7f936ff2b700 1 --2- 172.21.15.46:0/2078712460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9368105020 0x7f936810b8d0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:30.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.069+0000 7f936ff2b700 1 --2- 172.21.15.46:0/2078712460 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9368103480 0x7f93681038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:30.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.069+0000 7f936ff2b700 1 --2- 172.21.15.46:0/2078712460 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9368104680 0x7f9368104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:30.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.069+0000 7f936ff2b700 1 -- 172.21.15.46:0/2078712460 >> 172.21.15.46:0/2078712460 conn(0x7f93680fea20 msgr2=0x7f9368100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:30.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.069+0000 7f936ff2b700 1 -- 172.21.15.46:0/2078712460 shutdown_connections 2024-04-26T21:34:30.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.069+0000 7f936ff2b700 1 -- 172.21.15.46:0/2078712460 wait complete. 2024-04-26T21:34:30.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.070+0000 7f936ff2b700 1 Processor -- start 2024-04-26T21:34:30.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.070+0000 7f936ff2b700 1 -- start start 2024-04-26T21:34:30.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.070+0000 7f936ff2b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9368103480 0x7f936819b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:30.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.070+0000 7f936ff2b700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9368104680 0x7f936819bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:30.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.070+0000 7f936ff2b700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9368105020 0x7f93681a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:30.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.070+0000 7f936ff2b700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f936810ef00 con 0x7f9368104680 2024-04-26T21:34:30.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.070+0000 7f936ff2b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f936810ed80 con 0x7f9368103480 2024-04-26T21:34:30.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.070+0000 7f936ff2b700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f936810f080 con 0x7f9368105020 2024-04-26T21:34:30.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.070+0000 7f936d4c6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9368104680 0x7f936819bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:30.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.070+0000 7f936d4c6700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9368104680 0x7f936819bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48596/0 (socket says 172.21.15.46:48596) 2024-04-26T21:34:30.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.070+0000 7f936d4c6700 1 -- 172.21.15.46:0/3244108219 learned_addr learned my addr 172.21.15.46:0/3244108219 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:30.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.070+0000 7f936e4c8700 1 --2- 172.21.15.46:0/3244108219 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9368105020 0x7f93681a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:30.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.070+0000 7f936dcc7700 1 --2- 172.21.15.46:0/3244108219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9368103480 0x7f936819b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:30.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.070+0000 7f936d4c6700 1 -- 172.21.15.46:0/3244108219 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9368105020 msgr2=0x7f93681a2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:30.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.070+0000 7f936d4c6700 1 --2- 172.21.15.46:0/3244108219 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9368105020 0x7f93681a2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:30.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.070+0000 7f936d4c6700 1 -- 172.21.15.46:0/3244108219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9368103480 msgr2=0x7f936819b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.070+0000 7f936d4c6700 1 --2- 172.21.15.46:0/3244108219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9368103480 0x7f936819b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.070+0000 7f936d4c6700 1 -- 172.21.15.46:0/3244108219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93681a3360 con 0x7f9368104680 2024-04-26T21:34:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.071+0000 7f936d4c6700 1 --2- 172.21.15.46:0/3244108219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9368104680 0x7f936819bc40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f935c00d980 tx=0x7f935c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.071+0000 7f9366ffd700 1 -- 172.21.15.46:0/3244108219 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935c016070 con 0x7f9368104680 2024-04-26T21:34:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.071+0000 7f936ff2b700 1 -- 172.21.15.46:0/3244108219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93681a3650 con 0x7f9368104680 2024-04-26T21:34:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.071+0000 7f9366ffd700 1 -- 172.21.15.46:0/3244108219 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f935c00de70 con 0x7f9368104680 2024-04-26T21:34:30.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.071+0000 7f9366ffd700 1 -- 172.21.15.46:0/3244108219 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935c00f3c0 con 0x7f9368104680 2024-04-26T21:34:30.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.071+0000 7f936ff2b700 1 -- 172.21.15.46:0/3244108219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93681a3be0 con 0x7f9368104680 2024-04-26T21:34:30.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.072+0000 7f936ff2b700 1 -- 172.21.15.46:0/3244108219 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f936806a110 con 0x7f9368104680 2024-04-26T21:34:30.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.073+0000 7f9366ffd700 1 -- 172.21.15.46:0/3244108219 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f935c019070 con 0x7f9368104680 2024-04-26T21:34:30.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.074+0000 7f9366ffd700 1 --2- 172.21.15.46:0/3244108219 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f93500a3d50 0x7f93500a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:30.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.074+0000 7f936dcc7700 1 --2- 172.21.15.46:0/3244108219 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f93500a3d50 0x7f93500a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:30.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.074+0000 7f9366ffd700 1 -- 172.21.15.46:0/3244108219 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f935c0cfff0 con 0x7f9368104680 2024-04-26T21:34:30.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.074+0000 7f936dcc7700 1 --2- 172.21.15.46:0/3244108219 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f93500a3d50 0x7f93500a6210 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9360005fd0 tx=0x7f9360005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:30.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.078+0000 7f9366ffd700 1 -- 172.21.15.46:0/3244108219 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f935c09c8a0 con 0x7f9368104680 2024-04-26T21:34:30.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:29 smithi050 ceph-mon[32753]: pgmap v346: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:30.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.295+0000 7f936ff2b700 1 -- 172.21.15.46:0/3244108219 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f9368076d70 con 0x7f93500a3d50 2024-04-26T21:34:30.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.297+0000 7f9366ffd700 1 -- 172.21.15.46:0/3244108219 <== mgr.14150 v2:172.21.15.46:6800/4004175376 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f9368076d70 con 0x7f93500a3d50 2024-04-26T21:34:30.297 INFO:teuthology.orchestra.run.smithi046.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-26T21:34:30.297 INFO:teuthology.orchestra.run.smithi046.stdout:smithi046 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7512009A480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-04-26T21:34:30.297 INFO:teuthology.orchestra.run.smithi046.stdout:smithi050 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513001F480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-04-26T21:34:30.297 INFO:teuthology.orchestra.run.smithi046.stdout:smithi142 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6303009Y400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-26T21:34:30.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.299+0000 7f936ff2b700 1 -- 172.21.15.46:0/3244108219 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f93500a3d50 msgr2=0x7f93500a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:30.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.299+0000 7f936ff2b700 1 --2- 172.21.15.46:0/3244108219 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f93500a3d50 0x7f93500a6210 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9360005fd0 tx=0x7f9360005e50 comp rx=0 tx=0).stop 2024-04-26T21:34:30.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.299+0000 7f936ff2b700 1 -- 172.21.15.46:0/3244108219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9368104680 msgr2=0x7f936819bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:30.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.299+0000 7f936ff2b700 1 --2- 172.21.15.46:0/3244108219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9368104680 0x7f936819bc40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f935c00d980 tx=0x7f935c00dc90 comp rx=0 tx=0).stop 2024-04-26T21:34:30.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.299+0000 7f936ff2b700 1 -- 172.21.15.46:0/3244108219 shutdown_connections 2024-04-26T21:34:30.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.299+0000 7f936ff2b700 1 --2- 172.21.15.46:0/3244108219 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9368105020 0x7f93681a2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:30.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.299+0000 7f936ff2b700 1 --2- 172.21.15.46:0/3244108219 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9368103480 0x7f936819b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:30.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.299+0000 7f936ff2b700 1 --2- 172.21.15.46:0/3244108219 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f93500a3d50 0x7f93500a6210 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:30.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.299+0000 7f936ff2b700 1 --2- 172.21.15.46:0/3244108219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9368104680 0x7f936819bc40 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:30.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.299+0000 7f936ff2b700 1 -- 172.21.15.46:0/3244108219 >> 172.21.15.46:0/3244108219 conn(0x7f93680fea20 msgr2=0x7f9368075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:30.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.299+0000 7f936ff2b700 1 -- 172.21.15.46:0/3244108219 shutdown_connections 2024-04-26T21:34:30.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:30.299+0000 7f936ff2b700 1 -- 172.21.15.46:0/3244108219 wait complete. 2024-04-26T21:34:31.027 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-26T21:34:31.036 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi046.front.sepia.ceph.com 2024-04-26T21:34:31.036 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-26T21:34:31.287 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:34:31.671 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:31 smithi046 ceph-mon[29201]: pgmap v347: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:34:31.671 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:31 smithi046 ceph-mon[29201]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:34:32.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:31 smithi142 ceph-mon[32944]: pgmap v347: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:34:32.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:31 smithi142 ceph-mon[32944]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:34:32.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:31 smithi050 ceph-mon[32753]: pgmap v347: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:34:32.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:31 smithi050 ceph-mon[32753]: from='client.15150 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:34:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.131+0000 7f8511725700 1 -- 172.21.15.46:0/4121653033 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f850c103f40 msgr2=0x7f850c10a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.131+0000 7f8511725700 1 --2- 172.21.15.46:0/4121653033 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f850c103f40 0x7f850c10a7f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f850400b0d0 tx=0x7f850400b3e0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.131+0000 7f8511725700 1 -- 172.21.15.46:0/4121653033 shutdown_connections 2024-04-26T21:34:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.131+0000 7f8511725700 1 --2- 172.21.15.46:0/4121653033 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f850c103f40 0x7f850c10a7f0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.131+0000 7f8511725700 1 --2- 172.21.15.46:0/4121653033 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f850c1023a0 0x7f850c1027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.131+0000 7f8511725700 1 --2- 172.21.15.46:0/4121653033 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f850c1035a0 0x7f850c103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.131+0000 7f8511725700 1 -- 172.21.15.46:0/4121653033 >> 172.21.15.46:0/4121653033 conn(0x7f850c0fd940 msgr2=0x7f850c0ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.131+0000 7f8511725700 1 -- 172.21.15.46:0/4121653033 shutdown_connections 2024-04-26T21:34:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.131+0000 7f8511725700 1 -- 172.21.15.46:0/4121653033 wait complete. 2024-04-26T21:34:33.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.131+0000 7f8511725700 1 Processor -- start 2024-04-26T21:34:33.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.132+0000 7f8511725700 1 -- start start 2024-04-26T21:34:33.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.132+0000 7f8511725700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f850c1023a0 0x7f850c06c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:33.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.132+0000 7f8511725700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f850c1035a0 0x7f850c19e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:33.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.132+0000 7f8511725700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f850c103f40 0x7f850c1a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:33.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.132+0000 7f8511725700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f850c10de40 con 0x7f850c1023a0 2024-04-26T21:34:33.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.132+0000 7f8511725700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f850c10dcc0 con 0x7f850c103f40 2024-04-26T21:34:33.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.132+0000 7f8511725700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f850c10dfc0 con 0x7f850c1035a0 2024-04-26T21:34:33.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.132+0000 7f850affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f850c1023a0 0x7f850c06c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:33.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.132+0000 7f850affd700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f850c1023a0 0x7f850c06c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48616/0 (socket says 172.21.15.46:48616) 2024-04-26T21:34:33.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.132+0000 7f850affd700 1 -- 172.21.15.46:0/3670661485 learned_addr learned my addr 172.21.15.46:0/3670661485 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:33.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.132+0000 7f850a7fc700 1 --2- 172.21.15.46:0/3670661485 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f850c1035a0 0x7f850c19e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.132+0000 7f850b7fe700 1 --2- 172.21.15.46:0/3670661485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f850c103f40 0x7f850c1a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.132+0000 7f850affd700 1 -- 172.21.15.46:0/3670661485 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f850c1035a0 msgr2=0x7f850c19e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.132+0000 7f850affd700 1 --2- 172.21.15.46:0/3670661485 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f850c1035a0 0x7f850c19e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.132+0000 7f850affd700 1 -- 172.21.15.46:0/3670661485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f850c103f40 msgr2=0x7f850c1a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.132+0000 7f850affd700 1 --2- 172.21.15.46:0/3670661485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f850c103f40 0x7f850c1a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.133+0000 7f850affd700 1 -- 172.21.15.46:0/3670661485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f850c1a10d0 con 0x7f850c1023a0 2024-04-26T21:34:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.133+0000 7f850affd700 1 --2- 172.21.15.46:0/3670661485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f850c1023a0 0x7f850c06c6b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f84fc00c8f0 tx=0x7f84fc00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.133+0000 7f84f3fff700 1 -- 172.21.15.46:0/3670661485 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84fc004220 con 0x7f850c1023a0 2024-04-26T21:34:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.133+0000 7f84f3fff700 1 -- 172.21.15.46:0/3670661485 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84fc0043c0 con 0x7f850c1023a0 2024-04-26T21:34:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.133+0000 7f8511725700 1 -- 172.21.15.46:0/3670661485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f850c1a13c0 con 0x7f850c1023a0 2024-04-26T21:34:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.133+0000 7f84f3fff700 1 -- 172.21.15.46:0/3670661485 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84fc003c60 con 0x7f850c1023a0 2024-04-26T21:34:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.133+0000 7f8511725700 1 -- 172.21.15.46:0/3670661485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f850c1a18b0 con 0x7f850c1023a0 2024-04-26T21:34:33.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.134+0000 7f8511725700 1 -- 172.21.15.46:0/3670661485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f850c06a110 con 0x7f850c1023a0 2024-04-26T21:34:33.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.135+0000 7f84f3fff700 1 -- 172.21.15.46:0/3670661485 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f84fc007c80 con 0x7f850c1023a0 2024-04-26T21:34:33.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.136+0000 7f84f3fff700 1 --2- 172.21.15.46:0/3670661485 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f84f40a3cd0 0x7f84f40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:33.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.136+0000 7f850a7fc700 1 --2- 172.21.15.46:0/3670661485 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f84f40a3cd0 0x7f84f40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:33.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.136+0000 7f84f3fff700 1 -- 172.21.15.46:0/3670661485 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f84fc0cfd80 con 0x7f850c1023a0 2024-04-26T21:34:33.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.136+0000 7f850a7fc700 1 --2- 172.21.15.46:0/3670661485 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f84f40a3cd0 0x7f84f40a6190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f850c06cc30 tx=0x7f850000a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:33.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.140+0000 7f84f3fff700 1 -- 172.21.15.46:0/3670661485 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f84fc014030 con 0x7f850c1023a0 2024-04-26T21:34:33.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.394+0000 7f8511725700 1 -- 172.21.15.46:0/3670661485 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f850c02e4c0 con 0x7f850c1023a0 2024-04-26T21:34:33.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.395+0000 7f84f3fff700 1 -- 172.21.15.46:0/3670661485 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) ==== 76+0+3044 (secure 0 0 0) 0x7f84fc09c630 con 0x7f850c1023a0 2024-04-26T21:34:33.396 INFO:teuthology.orchestra.run.smithi046.stdout:e22 2024-04-26T21:34:33.396 INFO:teuthology.orchestra.run.smithi046.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T21:34:33.396 INFO:teuthology.orchestra.run.smithi046.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T21:34:33.396 INFO:teuthology.orchestra.run.smithi046.stdout:legacy client fscid: 1 2024-04-26T21:34:33.396 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:34:33.396 INFO:teuthology.orchestra.run.smithi046.stdout:Filesystem 'cephfs' (1) 2024-04-26T21:34:33.397 INFO:teuthology.orchestra.run.smithi046.stdout:fs_name cephfs 2024-04-26T21:34:33.397 INFO:teuthology.orchestra.run.smithi046.stdout:epoch 22 2024-04-26T21:34:33.397 INFO:teuthology.orchestra.run.smithi046.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-26T21:34:33.397 INFO:teuthology.orchestra.run.smithi046.stdout:created 2024-04-26T21:32:44.201541+0000 2024-04-26T21:34:33.397 INFO:teuthology.orchestra.run.smithi046.stdout:modified 2024-04-26T21:32:49.889029+0000 2024-04-26T21:34:33.397 INFO:teuthology.orchestra.run.smithi046.stdout:tableserver 0 2024-04-26T21:34:33.397 INFO:teuthology.orchestra.run.smithi046.stdout:root 0 2024-04-26T21:34:33.397 INFO:teuthology.orchestra.run.smithi046.stdout:session_timeout 300 2024-04-26T21:34:33.397 INFO:teuthology.orchestra.run.smithi046.stdout:session_autoclose 300 2024-04-26T21:34:33.397 INFO:teuthology.orchestra.run.smithi046.stdout:max_file_size 1099511627776 2024-04-26T21:34:33.397 INFO:teuthology.orchestra.run.smithi046.stdout:max_xattr_size 65536 2024-04-26T21:34:33.397 INFO:teuthology.orchestra.run.smithi046.stdout:required_client_features {} 2024-04-26T21:34:33.397 INFO:teuthology.orchestra.run.smithi046.stdout:last_failure 0 2024-04-26T21:34:33.397 INFO:teuthology.orchestra.run.smithi046.stdout:last_failure_osd_epoch 0 2024-04-26T21:34:33.397 INFO:teuthology.orchestra.run.smithi046.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T21:34:33.398 INFO:teuthology.orchestra.run.smithi046.stdout:max_mds 3 2024-04-26T21:34:33.398 INFO:teuthology.orchestra.run.smithi046.stdout:in 0,1,2 2024-04-26T21:34:33.398 INFO:teuthology.orchestra.run.smithi046.stdout:up {0=24539,1=24526,2=24509} 2024-04-26T21:34:33.398 INFO:teuthology.orchestra.run.smithi046.stdout:failed 2024-04-26T21:34:33.398 INFO:teuthology.orchestra.run.smithi046.stdout:damaged 2024-04-26T21:34:33.398 INFO:teuthology.orchestra.run.smithi046.stdout:stopped 2024-04-26T21:34:33.398 INFO:teuthology.orchestra.run.smithi046.stdout:data_pools [3] 2024-04-26T21:34:33.398 INFO:teuthology.orchestra.run.smithi046.stdout:metadata_pool 2 2024-04-26T21:34:33.398 INFO:teuthology.orchestra.run.smithi046.stdout:inline_data disabled 2024-04-26T21:34:33.398 INFO:teuthology.orchestra.run.smithi046.stdout:balancer 2024-04-26T21:34:33.398 INFO:teuthology.orchestra.run.smithi046.stdout:bal_rank_mask -1 2024-04-26T21:34:33.398 INFO:teuthology.orchestra.run.smithi046.stdout:standby_count_wanted 0 2024-04-26T21:34:33.398 INFO:teuthology.orchestra.run.smithi046.stdout:qdb_cluster leader: 24539 members: 24509,24526,24539 2024-04-26T21:34:33.398 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.b{0:24539} state up:active seq 3 addr [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T21:34:33.399 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.c{0:24505} state up:standby-replay seq 1 addr [v2:172.21.15.142:6836/1723132553,v1:172.21.15.142:6837/1723132553] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T21:34:33.399 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.i{1:24526} state up:active seq 6 addr [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T21:34:33.399 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.f{1:24493} state up:standby-replay seq 1 addr [v2:172.21.15.142:6834/3791468066,v1:172.21.15.142:6835/3791468066] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T21:34:33.399 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.h{2:24509} state up:active seq 7 addr [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T21:34:33.399 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.e{2:24470} state up:standby-replay seq 1 addr [v2:172.21.15.50:6834/2546509863,v1:172.21.15.50:6835/2546509863] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T21:34:33.399 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:34:33.399 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:34:33.399 INFO:teuthology.orchestra.run.smithi046.stdout:Standby daemons: 2024-04-26T21:34:33.399 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:34:33.399 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.j{-1:14553} state up:standby seq 1 addr [v2:172.21.15.46:6834/3674296286,v1:172.21.15.46:6835/3674296286] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T21:34:33.399 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.d{-1:14565} state up:standby seq 1 addr [v2:172.21.15.46:6836/391791206,v1:172.21.15.46:6837/391791206] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T21:34:33.399 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.g{-1:14640} state up:standby seq 1 addr [v2:172.21.15.46:6838/1222489912,v1:172.21.15.46:6839/1222489912] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T21:34:33.399 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.a{-1:14658} state up:standby seq 1 addr [v2:172.21.15.46:6840/860605548,v1:172.21.15.46:6841/860605548] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T21:34:33.399 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.l{-1:24442} state up:standby seq 1 addr [v2:172.21.15.142:6832/3621399933,v1:172.21.15.142:6833/3621399933] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T21:34:33.399 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.k{-1:24446} state up:standby seq 1 addr [v2:172.21.15.50:6832/345015778,v1:172.21.15.50:6833/345015778] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T21:34:33.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.397+0000 7f8511725700 1 -- 172.21.15.46:0/3670661485 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f84f40a3cd0 msgr2=0x7f84f40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:33.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.397+0000 7f8511725700 1 --2- 172.21.15.46:0/3670661485 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f84f40a3cd0 0x7f84f40a6190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f850c06cc30 tx=0x7f850000a380 comp rx=0 tx=0).stop 2024-04-26T21:34:33.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.397+0000 7f8511725700 1 -- 172.21.15.46:0/3670661485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f850c1023a0 msgr2=0x7f850c06c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:33.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.397+0000 7f8511725700 1 --2- 172.21.15.46:0/3670661485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f850c1023a0 0x7f850c06c6b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f84fc00c8f0 tx=0x7f84fc00cc00 comp rx=0 tx=0).stop 2024-04-26T21:34:33.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.397+0000 7f8511725700 1 -- 172.21.15.46:0/3670661485 shutdown_connections 2024-04-26T21:34:33.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.397+0000 7f8511725700 1 --2- 172.21.15.46:0/3670661485 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f84f40a3cd0 0x7f84f40a6190 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.397+0000 7f8511725700 1 --2- 172.21.15.46:0/3670661485 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f850c1023a0 0x7f850c06c6b0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.397+0000 7f8511725700 1 --2- 172.21.15.46:0/3670661485 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f850c103f40 0x7f850c1a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.398+0000 7f8511725700 1 --2- 172.21.15.46:0/3670661485 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f850c1035a0 0x7f850c19e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.398+0000 7f8511725700 1 -- 172.21.15.46:0/3670661485 >> 172.21.15.46:0/3670661485 conn(0x7f850c0fd940 msgr2=0x7f850c1067d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:33.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.398+0000 7f8511725700 1 -- 172.21.15.46:0/3670661485 shutdown_connections 2024-04-26T21:34:33.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:33.398+0000 7f8511725700 1 -- 172.21.15.46:0/3670661485 wait complete. 2024-04-26T21:34:33.401 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 22 2024-04-26T21:34:33.777 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:33 smithi046 ceph-mon[29201]: pgmap v348: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:33.778 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:33 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3670661485' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T21:34:34.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:33 smithi142 ceph-mon[32944]: pgmap v348: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:34.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:33 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3670661485' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T21:34:34.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:33 smithi050 ceph-mon[32753]: pgmap v348: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:34.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:33 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3670661485' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T21:34:34.195 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-26T21:34:34.446 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:34:35.906 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:35 smithi046 ceph-mon[29201]: pgmap v349: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:36.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:35 smithi142 ceph-mon[32944]: pgmap v349: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:36.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:35 smithi050 ceph-mon[32753]: pgmap v349: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:36.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.331+0000 7fadb4856700 1 -- 172.21.15.46:0/1439270918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb0101d20 msgr2=0x7fadb0104150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:36.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.331+0000 7fadb4856700 1 --2- 172.21.15.46:0/1439270918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb0101d20 0x7fadb0104150 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fada4009990 tx=0x7fada4009ca0 comp rx=0 tx=0).stop 2024-04-26T21:34:36.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.331+0000 7fadb4856700 1 -- 172.21.15.46:0/1439270918 shutdown_connections 2024-04-26T21:34:36.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.331+0000 7fadb4856700 1 --2- 172.21.15.46:0/1439270918 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb0101d20 0x7fadb0104150 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:36.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.331+0000 7fadb4856700 1 --2- 172.21.15.46:0/1439270918 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadb0107060 0x7fadb01074e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:36.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.331+0000 7fadb4856700 1 --2- 172.21.15.46:0/1439270918 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fadb0104690 0x7fadb0106b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:36.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.331+0000 7fadb4856700 1 -- 172.21.15.46:0/1439270918 >> 172.21.15.46:0/1439270918 conn(0x7fadb00fd980 msgr2=0x7fadb00ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:36.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.331+0000 7fadb4856700 1 -- 172.21.15.46:0/1439270918 shutdown_connections 2024-04-26T21:34:36.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.331+0000 7fadb4856700 1 -- 172.21.15.46:0/1439270918 wait complete. 2024-04-26T21:34:36.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.332+0000 7fadb4856700 1 Processor -- start 2024-04-26T21:34:36.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.332+0000 7fadb4856700 1 -- start start 2024-04-26T21:34:36.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.332+0000 7fadb4856700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb0101d20 0x7fadb019fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:36.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.332+0000 7fadb4856700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fadb0104690 0x7fadb01a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:36.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.332+0000 7fadb4856700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadb0107060 0x7fadb01a7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:36.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.332+0000 7fadb4856700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fadb0115510 con 0x7fadb0101d20 2024-04-26T21:34:36.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.332+0000 7fadb4856700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fadb0115390 con 0x7fadb0104690 2024-04-26T21:34:36.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.332+0000 7fadb4856700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fadb0115690 con 0x7fadb0107060 2024-04-26T21:34:36.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.332+0000 7fadae59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb0101d20 0x7fadb019fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:36.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.332+0000 7fadae59c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb0101d20 0x7fadb019fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41776/0 (socket says 172.21.15.46:41776) 2024-04-26T21:34:36.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.332+0000 7fadae59c700 1 -- 172.21.15.46:0/403081215 learned_addr learned my addr 172.21.15.46:0/403081215 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:36.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.332+0000 7fadadd9b700 1 --2- 172.21.15.46:0/403081215 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fadb0104690 0x7fadb01a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:36.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.332+0000 7fadaed9d700 1 --2- 172.21.15.46:0/403081215 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadb0107060 0x7fadb01a7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:36.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.332+0000 7fadae59c700 1 -- 172.21.15.46:0/403081215 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadb0107060 msgr2=0x7fadb01a7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:36.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.332+0000 7fadae59c700 1 --2- 172.21.15.46:0/403081215 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadb0107060 0x7fadb01a7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:36.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.332+0000 7fadae59c700 1 -- 172.21.15.46:0/403081215 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fadb0104690 msgr2=0x7fadb01a0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:36.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.332+0000 7fadae59c700 1 --2- 172.21.15.46:0/403081215 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fadb0104690 0x7fadb01a0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:36.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.332+0000 7fadae59c700 1 -- 172.21.15.46:0/403081215 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadb01a7860 con 0x7fadb0101d20 2024-04-26T21:34:36.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.332+0000 7fadae59c700 1 --2- 172.21.15.46:0/403081215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb0101d20 0x7fadb019fc10 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fada4006010 tx=0x7fada40057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:36.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.333+0000 7fad9f7fe700 1 -- 172.21.15.46:0/403081215 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada4027070 con 0x7fadb0101d20 2024-04-26T21:34:36.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.333+0000 7fadb4856700 1 -- 172.21.15.46:0/403081215 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadb01a7af0 con 0x7fadb0101d20 2024-04-26T21:34:36.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.333+0000 7fad9f7fe700 1 -- 172.21.15.46:0/403081215 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fada4004140 con 0x7fadb0101d20 2024-04-26T21:34:36.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.333+0000 7fad9f7fe700 1 -- 172.21.15.46:0/403081215 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada4022e40 con 0x7fadb0101d20 2024-04-26T21:34:36.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.333+0000 7fadb4856700 1 -- 172.21.15.46:0/403081215 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadb01a8050 con 0x7fadb0101d20 2024-04-26T21:34:36.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.334+0000 7fadb4856700 1 -- 172.21.15.46:0/403081215 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadb002e490 con 0x7fadb0101d20 2024-04-26T21:34:36.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.335+0000 7fad9f7fe700 1 -- 172.21.15.46:0/403081215 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fada402a070 con 0x7fadb0101d20 2024-04-26T21:34:36.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.335+0000 7fad9f7fe700 1 --2- 172.21.15.46:0/403081215 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fad980a3cd0 0x7fad980a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:36.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.336+0000 7fadadd9b700 1 --2- 172.21.15.46:0/403081215 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fad980a3cd0 0x7fad980a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:36.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.336+0000 7fad9f7fe700 1 -- 172.21.15.46:0/403081215 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7fada40d1300 con 0x7fadb0101d20 2024-04-26T21:34:36.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.336+0000 7fadadd9b700 1 --2- 172.21.15.46:0/403081215 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fad980a3cd0 0x7fad980a6190 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fada0005950 tx=0x7fada000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:36.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.340+0000 7fad9f7fe700 1 -- 172.21.15.46:0/403081215 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fada4013070 con 0x7fadb0101d20 2024-04-26T21:34:36.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.548+0000 7fadb4856700 1 -- 172.21.15.46:0/403081215 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fadb002e840 con 0x7fadb0101d20 2024-04-26T21:34:36.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.549+0000 7fad9f7fe700 1 -- 172.21.15.46:0/403081215 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v87) ==== 56+0+5256 (secure 0 0 0) 0x7fada409dbe0 con 0x7fadb0101d20 2024-04-26T21:34:36.550 INFO:teuthology.orchestra.run.smithi046.stdout:epoch 87 2024-04-26T21:34:36.550 INFO:teuthology.orchestra.run.smithi046.stdout:fsid 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:34:36.550 INFO:teuthology.orchestra.run.smithi046.stdout:created 2024-04-26T21:24:30.481156+0000 2024-04-26T21:34:36.550 INFO:teuthology.orchestra.run.smithi046.stdout:modified 2024-04-26T21:33:31.051554+0000 2024-04-26T21:34:36.550 INFO:teuthology.orchestra.run.smithi046.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-26T21:34:36.550 INFO:teuthology.orchestra.run.smithi046.stdout:crush_version 26 2024-04-26T21:34:36.550 INFO:teuthology.orchestra.run.smithi046.stdout:full_ratio 0.95 2024-04-26T21:34:36.550 INFO:teuthology.orchestra.run.smithi046.stdout:backfillfull_ratio 0.9 2024-04-26T21:34:36.550 INFO:teuthology.orchestra.run.smithi046.stdout:nearfull_ratio 0.85 2024-04-26T21:34:36.550 INFO:teuthology.orchestra.run.smithi046.stdout:require_min_compat_client luminous 2024-04-26T21:34:36.550 INFO:teuthology.orchestra.run.smithi046.stdout:min_compat_client luminous 2024-04-26T21:34:36.550 INFO:teuthology.orchestra.run.smithi046.stdout:require_osd_release squid 2024-04-26T21:34:36.551 INFO:teuthology.orchestra.run.smithi046.stdout:stretch_mode_enabled false 2024-04-26T21:34:36.551 INFO:teuthology.orchestra.run.smithi046.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-26T21:34:36.551 INFO:teuthology.orchestra.run.smithi046.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-26T21:34:36.551 INFO:teuthology.orchestra.run.smithi046.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-26T21:34:36.551 INFO:teuthology.orchestra.run.smithi046.stdout:max_osd 12 2024-04-26T21:34:36.551 INFO:teuthology.orchestra.run.smithi046.stdout:osd.0 up in weight 1 up_from 8 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.46:6802/2762665125,v1:172.21.15.46:6803/2762665125] [v2:172.21.15.46:6804/2762665125,v1:172.21.15.46:6805/2762665125] exists,up f53c8702-e4da-435f-a9dc-3911d0d36ef5 2024-04-26T21:34:36.551 INFO:teuthology.orchestra.run.smithi046.stdout:osd.1 up in weight 1 up_from 13 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6800/1628008970,v1:172.21.15.50:6801/1628008970] [v2:172.21.15.50:6802/1628008970,v1:172.21.15.50:6803/1628008970] exists,up fc4d6f52-fc8a-4efd-80c9-c092b57e7f42 2024-04-26T21:34:36.551 INFO:teuthology.orchestra.run.smithi046.stdout:osd.2 up in weight 1 up_from 18 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.142:6800/1513279137,v1:172.21.15.142:6801/1513279137] [v2:172.21.15.142:6802/1513279137,v1:172.21.15.142:6803/1513279137] exists,up 26d1baa1-4ea1-4b7e-9fcf-4f8bbf7329c5 2024-04-26T21:34:36.551 INFO:teuthology.orchestra.run.smithi046.stdout:osd.3 up in weight 1 up_from 25 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.46:6810/2435181760,v1:172.21.15.46:6811/2435181760] [v2:172.21.15.46:6812/2435181760,v1:172.21.15.46:6813/2435181760] exists,up 34d2401f-5fcc-491c-8fcf-c3c492611f23 2024-04-26T21:34:36.551 INFO:teuthology.orchestra.run.smithi046.stdout:osd.4 up in weight 1 up_from 33 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6808/3177333250,v1:172.21.15.50:6809/3177333250] [v2:172.21.15.50:6810/3177333250,v1:172.21.15.50:6811/3177333250] exists,up 986bb709-e4cf-400d-982e-c3f5ac413e0e 2024-04-26T21:34:36.551 INFO:teuthology.orchestra.run.smithi046.stdout:osd.5 up in weight 1 up_from 38 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.142:6808/4113107737,v1:172.21.15.142:6809/4113107737] [v2:172.21.15.142:6810/4113107737,v1:172.21.15.142:6811/4113107737] exists,up b7d65851-ebe1-45e2-833d-8022446f2698 2024-04-26T21:34:36.551 INFO:teuthology.orchestra.run.smithi046.stdout:osd.6 up in weight 1 up_from 46 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.46:6818/2325162882,v1:172.21.15.46:6819/2325162882] [v2:172.21.15.46:6820/2325162882,v1:172.21.15.46:6821/2325162882] exists,up 2a3e96d7-d5bf-44ff-9134-29f1420c0e4c 2024-04-26T21:34:36.551 INFO:teuthology.orchestra.run.smithi046.stdout:osd.7 up in weight 1 up_from 51 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6816/820310228,v1:172.21.15.50:6817/820310228] [v2:172.21.15.50:6818/820310228,v1:172.21.15.50:6819/820310228] exists,up e53ae885-23b6-46fd-b4b1-c62f31a6921f 2024-04-26T21:34:36.551 INFO:teuthology.orchestra.run.smithi046.stdout:osd.8 up in weight 1 up_from 59 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.142:6816/1824918711,v1:172.21.15.142:6817/1824918711] [v2:172.21.15.142:6818/1824918711,v1:172.21.15.142:6819/1824918711] exists,up 356e2692-16a9-4045-b58f-c8f9cb5d7241 2024-04-26T21:34:36.552 INFO:teuthology.orchestra.run.smithi046.stdout:osd.9 up in weight 1 up_from 67 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.46:6826/3601947736,v1:172.21.15.46:6827/3601947736] [v2:172.21.15.46:6828/3601947736,v1:172.21.15.46:6829/3601947736] exists,up 8e55570e-77e5-4892-8b9f-77dcd03a4eec 2024-04-26T21:34:36.552 INFO:teuthology.orchestra.run.smithi046.stdout:osd.10 up in weight 1 up_from 72 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6824/3135765006,v1:172.21.15.50:6825/3135765006] [v2:172.21.15.50:6826/3135765006,v1:172.21.15.50:6827/3135765006] exists,up 366389fa-fb9a-408b-b5c2-343c054c5a64 2024-04-26T21:34:36.552 INFO:teuthology.orchestra.run.smithi046.stdout:osd.11 up in weight 1 up_from 80 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.142:6824/1482712612,v1:172.21.15.142:6825/1482712612] [v2:172.21.15.142:6826/1482712612,v1:172.21.15.142:6827/1482712612] exists,up 7b6e3d93-59e1-4853-b4a2-261b728c47a5 2024-04-26T21:34:36.552 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 2.a [2,11] 2024-04-26T21:34:36.552 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 2.b [8,11] 2024-04-26T21:34:36.552 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 2.10 [2,4] 2024-04-26T21:34:36.552 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 3.14 [4,6] 2024-04-26T21:34:36.552 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 3.1d [4,8,5,9] 2024-04-26T21:34:36.552 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 3.24 [4,9] 2024-04-26T21:34:36.552 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 3.27 [3,9] 2024-04-26T21:34:36.552 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:0/2465242088 expires 2024-04-27T21:25:29.848343+0000 2024-04-26T21:34:36.552 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:6801/2053782984 expires 2024-04-27T21:25:29.848343+0000 2024-04-26T21:34:36.552 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:6800/2053782984 expires 2024-04-27T21:25:29.848343+0000 2024-04-26T21:34:36.552 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:0/1381997090 expires 2024-04-27T21:25:02.078730+0000 2024-04-26T21:34:36.552 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:0/3507268217 expires 2024-04-27T21:25:29.848343+0000 2024-04-26T21:34:36.552 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:0/2570024356 expires 2024-04-27T21:25:02.078730+0000 2024-04-26T21:34:36.553 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:0/2691061938 expires 2024-04-27T21:25:29.848343+0000 2024-04-26T21:34:36.553 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:0/2775954801 expires 2024-04-27T21:25:02.078730+0000 2024-04-26T21:34:36.553 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:6800/2896098005 expires 2024-04-27T21:25:02.078730+0000 2024-04-26T21:34:36.553 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:6801/2896098005 expires 2024-04-27T21:25:02.078730+0000 2024-04-26T21:34:36.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.551+0000 7fadb4856700 1 -- 172.21.15.46:0/403081215 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fad980a3cd0 msgr2=0x7fad980a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:36.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.551+0000 7fadb4856700 1 --2- 172.21.15.46:0/403081215 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fad980a3cd0 0x7fad980a6190 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fada0005950 tx=0x7fada000b410 comp rx=0 tx=0).stop 2024-04-26T21:34:36.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.551+0000 7fadb4856700 1 -- 172.21.15.46:0/403081215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb0101d20 msgr2=0x7fadb019fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:36.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.551+0000 7fadb4856700 1 --2- 172.21.15.46:0/403081215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb0101d20 0x7fadb019fc10 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fada4006010 tx=0x7fada40057c0 comp rx=0 tx=0).stop 2024-04-26T21:34:36.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.551+0000 7fadb4856700 1 -- 172.21.15.46:0/403081215 shutdown_connections 2024-04-26T21:34:36.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.551+0000 7fadb4856700 1 --2- 172.21.15.46:0/403081215 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fad980a3cd0 0x7fad980a6190 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:36.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.551+0000 7fadb4856700 1 --2- 172.21.15.46:0/403081215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fadb0101d20 0x7fadb019fc10 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:36.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.551+0000 7fadb4856700 1 --2- 172.21.15.46:0/403081215 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fadb0107060 0x7fadb01a7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:36.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.551+0000 7fadb4856700 1 --2- 172.21.15.46:0/403081215 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fadb0104690 0x7fadb01a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:36.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.551+0000 7fadb4856700 1 -- 172.21.15.46:0/403081215 >> 172.21.15.46:0/403081215 conn(0x7fadb00fd980 msgr2=0x7fadb01054c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:36.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.551+0000 7fadb4856700 1 -- 172.21.15.46:0/403081215 shutdown_connections 2024-04-26T21:34:36.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:36.551+0000 7fadb4856700 1 -- 172.21.15.46:0/403081215 wait complete. 2024-04-26T21:34:36.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:36 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/403081215' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T21:34:37.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:36 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/403081215' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T21:34:37.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:36 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/403081215' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T21:34:37.382 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-26T21:34:37.394 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:34:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.506+0000 7ff5badf0640 1 -- 172.21.15.46:0/1412223681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5b41033f0 msgr2=0x7ff5b41057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.506+0000 7ff5badf0640 1 --2- 172.21.15.46:0/1412223681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5b41033f0 0x7ff5b41057e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7ff59c009900 tx=0x7ff59c02a410 comp rx=0 tx=0).stop 2024-04-26T21:34:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.506+0000 7ff5badf0640 1 -- 172.21.15.46:0/1412223681 shutdown_connections 2024-04-26T21:34:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.506+0000 7ff5badf0640 1 --2- 172.21.15.46:0/1412223681 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff5b4108650 0x7ff5b410aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.506+0000 7ff5badf0640 1 --2- 172.21.15.46:0/1412223681 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5b4105d20 0x7ff5b4108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.506+0000 7ff5badf0640 1 --2- 172.21.15.46:0/1412223681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5b41033f0 0x7ff5b41057e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.506+0000 7ff5badf0640 1 -- 172.21.15.46:0/1412223681 >> 172.21.15.46:0/1412223681 conn(0x7ff5b40fcde0 msgr2=0x7ff5b40ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.506+0000 7ff5badf0640 1 -- 172.21.15.46:0/1412223681 shutdown_connections 2024-04-26T21:34:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.506+0000 7ff5badf0640 1 -- 172.21.15.46:0/1412223681 wait complete. 2024-04-26T21:34:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.506+0000 7ff5badf0640 1 Processor -- start 2024-04-26T21:34:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.506+0000 7ff5badf0640 1 -- start start 2024-04-26T21:34:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.507+0000 7ff5badf0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5b41033f0 0x7ff5b4198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.507+0000 7ff5badf0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5b4105d20 0x7ff5b41991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.507+0000 7ff5badf0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff5b4108650 0x7ff5b41a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.507+0000 7ff5badf0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b410e510 con 0x7ff5b41033f0 2024-04-26T21:34:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.507+0000 7ff5badf0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b410e390 con 0x7ff5b4105d20 2024-04-26T21:34:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.507+0000 7ff5badf0640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b410e690 con 0x7ff5b4108650 2024-04-26T21:34:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.507+0000 7ff5b8b8c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5b41033f0 0x7ff5b4198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.507+0000 7ff5b8b8c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5b41033f0 0x7ff5b4198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41794/0 (socket says 172.21.15.46:41794) 2024-04-26T21:34:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.507+0000 7ff5b8b8c640 1 -- 172.21.15.46:0/4256849860 learned_addr learned my addr 172.21.15.46:0/4256849860 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.507+0000 7ff5b3fff640 1 --2- 172.21.15.46:0/4256849860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5b4105d20 0x7ff5b41991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.507+0000 7ff5b938d640 1 --2- 172.21.15.46:0/4256849860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff5b4108650 0x7ff5b41a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.507+0000 7ff5b8b8c640 1 -- 172.21.15.46:0/4256849860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff5b4108650 msgr2=0x7ff5b41a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:37.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.507+0000 7ff5b8b8c640 1 --2- 172.21.15.46:0/4256849860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff5b4108650 0x7ff5b41a01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:37.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.507+0000 7ff5b8b8c640 1 -- 172.21.15.46:0/4256849860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5b4105d20 msgr2=0x7ff5b41991a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:37.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.507+0000 7ff5b8b8c640 1 --2- 172.21.15.46:0/4256849860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5b4105d20 0x7ff5b41991a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:37.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.507+0000 7ff5b8b8c640 1 -- 172.21.15.46:0/4256849860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5b41a0870 con 0x7ff5b41033f0 2024-04-26T21:34:37.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.507+0000 7ff5b938d640 1 --2- 172.21.15.46:0/4256849860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff5b4108650 0x7ff5b41a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:34:37.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.507+0000 7ff5b3fff640 1 --2- 172.21.15.46:0/4256849860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5b4105d20 0x7ff5b41991a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:34:37.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.507+0000 7ff5b8b8c640 1 --2- 172.21.15.46:0/4256849860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5b41033f0 0x7ff5b4198c60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff59c031500 tx=0x7ff59c0043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:37.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.507+0000 7ff5b1ffb640 1 -- 172.21.15.46:0/4256849860 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59c03f030 con 0x7ff5b41033f0 2024-04-26T21:34:37.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.507+0000 7ff5b1ffb640 1 -- 172.21.15.46:0/4256849860 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff59c003800 con 0x7ff5b41033f0 2024-04-26T21:34:37.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.507+0000 7ff5b1ffb640 1 -- 172.21.15.46:0/4256849860 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59c035070 con 0x7ff5b41033f0 2024-04-26T21:34:37.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.507+0000 7ff5badf0640 1 -- 172.21.15.46:0/4256849860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5b41a0b00 con 0x7ff5b41033f0 2024-04-26T21:34:37.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.508+0000 7ff5badf0640 1 -- 172.21.15.46:0/4256849860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5b41a0ff0 con 0x7ff5b41033f0 2024-04-26T21:34:37.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.509+0000 7ff5badf0640 1 -- 172.21.15.46:0/4256849860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5b40fe8d0 con 0x7ff5b41033f0 2024-04-26T21:34:37.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.510+0000 7ff5b1ffb640 1 -- 172.21.15.46:0/4256849860 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff59c0039b0 con 0x7ff5b41033f0 2024-04-26T21:34:37.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.511+0000 7ff5b1ffb640 1 --2- 172.21.15.46:0/4256849860 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ff5a00a3c40 0x7ff5a00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:37.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.511+0000 7ff5b3fff640 1 --2- 172.21.15.46:0/4256849860 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ff5a00a3c40 0x7ff5a00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:37.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.511+0000 7ff5b1ffb640 1 -- 172.21.15.46:0/4256849860 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7ff59c0f10d0 con 0x7ff5b41033f0 2024-04-26T21:34:37.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.511+0000 7ff5b3fff640 1 --2- 172.21.15.46:0/4256849860 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ff5a00a3c40 0x7ff5a00a6100 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff5a4005c40 tx=0x7ff5a400a430 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:37.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.515+0000 7ff5b1ffb640 1 -- 172.21.15.46:0/4256849860 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff59c0bd980 con 0x7ff5b41033f0 2024-04-26T21:34:37.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.705+0000 7ff5badf0640 1 -- 172.21.15.46:0/4256849860 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff5b402dd20 con 0x7ff5b41033f0 2024-04-26T21:34:37.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.707+0000 7ff5b1ffb640 1 -- 172.21.15.46:0/4256849860 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22602 (secure 0 0 0) 0x7ff59c0bd280 con 0x7ff5b41033f0 2024-04-26T21:34:37.708 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:34:37.708 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":87,"fsid":"36c7013a-0413-11ef-bc93-c7b262605968","created":"2024-04-26T21:24:30.481156+0000","modified":"2024-04-26T21:33:31.051554+0000","last_up_change":"2024-04-26T21:31:50.100491+0000","last_in_change":"2024-04-26T21:31:29.643428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:27:57.935988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:32:42.195211+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:32:43.162114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f53c8702-e4da-435f-a9dc-3911d0d36ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6803","nonce":2762665125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6805","nonce":2762665125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6809","nonce":2762665125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6807","nonce":2762665125}]},"public_addr":"172.21.15.46:6803/2762665125","cluster_addr":"172.21.15.46:6805/2762665125","heartbeat_back_addr":"172.21.15.46:6809/2762665125","heartbeat_front_addr":"172.21.15.46:6807/2762665125","state":["exists","up"]},{"osd":1,"uuid":"fc4d6f52-fc8a-4efd-80c9-c092b57e7f42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6801","nonce":1628008970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6803","nonce":1628008970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6807","nonce":1628008970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6805","nonce":1628008970}]},"public_addr":"172.21.15.50:6801/1628008970","cluster_addr":"172.21.15.50:6803/1628008970","heartbeat_back_addr":"172.21.15.50:6807/1628008970","heartbeat_front_addr":"172.21.15.50:6805/1628008970","state":["exists","up"]},{"osd":2,"uuid":"26d1baa1-4ea1-4b7e-9fcf-4f8bbf7329c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6801","nonce":1513279137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6803","nonce":1513279137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6807","nonce":1513279137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6805","nonce":1513279137}]},"public_addr":"172.21.15.142:6801/1513279137","cluster_addr":"172.21.15.142:6803/1513279137","heartbeat_back_addr":"172.21.15.142:6807/1513279137","heartbeat_front_addr":"172.21.15.142:6805/1513279137","state":["exists","up"]},{"osd":3,"uuid":"34d2401f-5fcc-491c-8fcf-c3c492611f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6811","nonce":2435181760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6813","nonce":2435181760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6817","nonce":2435181760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6815","nonce":2435181760}]},"public_addr":"172.21.15.46:6811/2435181760","cluster_addr":"172.21.15.46:6813/2435181760","heartbeat_back_addr":"172.21.15.46:6817/2435181760","heartbeat_front_addr":"172.21.15.46:6815/2435181760","state":["exists","up"]},{"osd":4,"uuid":"986bb709-e4cf-400d-982e-c3f5ac413e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6809","nonce":3177333250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6811","nonce":3177333250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6815","nonce":3177333250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6813","nonce":3177333250}]},"public_addr":"172.21.15.50:6809/3177333250","cluster_addr":"172.21.15.50:6811/3177333250","heartbeat_back_addr":"172.21.15.50:6815/3177333250","heartbeat_front_addr":"172.21.15.50:6813/3177333250","state":["exists","up"]},{"osd":5,"uuid":"b7d65851-ebe1-45e2-833d-8022446f2698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6809","nonce":4113107737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6811","nonce":4113107737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6815","nonce":4113107737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6813","nonce":4113107737}]},"public_addr":"172.21.15.142:6809/4113107737","cluster_addr":"172.21.15.142:6811/4113107737","heartbeat_back_addr":"172.21.15.142:6815/4113107737","heartbeat_front_addr":"172.21.15.142:6813/4113107737","state":["exists","up"]},{"osd":6,"uuid":"2a3e96d7-d5bf-44ff-9134-29f1420c0e4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6819","nonce":2325162882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6821","nonce":2325162882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6825","nonce":2325162882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6823","nonce":2325162882}]},"public_addr":"172.21.15.46:6819/2325162882","cluster_addr":"172.21.15.46:6821/2325162882","heartbeat_back_addr":"172.21.15.46:6825/2325162882","heartbeat_front_addr":"172.21.15.46:6823/2325162882","state":["exists","up"]},{"osd":7,"uuid":"e53ae885-23b6-46fd-b4b1-c62f31a6921f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6817","nonce":820310228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6819","nonce":820310228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6823","nonce":820310228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6821","nonce":820310228}]},"public_addr":"172.21.15.50:6817/820310228","cluster_addr":"172.21.15.50:6819/820310228","heartbeat_back_addr":"172.21.15.50:6823/820310228","heartbeat_front_addr":"172.21.15.50:6821/820310228","state":["exists","up"]},{"osd":8,"uuid":"356e2692-16a9-4045-b58f-c8f9cb5d7241","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6817","nonce":1824918711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6819","nonce":1824918711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6823","nonce":1824918711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6821","nonce":1824918711}]},"public_addr":"172.21.15.142:6817/1824918711","cluster_addr":"172.21.15.142:6819/1824918711","heartbeat_back_addr":"172.21.15.142:6823/1824918711","heartbeat_front_addr":"172.21.15.142:6821/1824918711","state":["exists","up"]},{"osd":9,"uuid":"8e55570e-77e5-4892-8b9f-77dcd03a4eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6827","nonce":3601947736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6829","nonce":3601947736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6833","nonce":3601947736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6831","nonce":3601947736}]},"public_addr":"172.21.15.46:6827/3601947736","cluster_addr":"172.21.15.46:6829/3601947736","heartbeat_back_addr":"172.21.15.46:6833/3601947736","heartbeat_front_addr":"172.21.15.46:6831/3601947736","state":["exists","up"]},{"osd":10,"uuid":"366389fa-fb9a-408b-b5c2-343c054c5a64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6825","nonce":3135765006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6827","nonce":3135765006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6831","nonce":3135765006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6829","nonce":3135765006}]},"public_addr":"172.21.15.50:6825/3135765006","cluster_addr":"172.21.15.50:6827/3135765006","heartbeat_back_addr":"172.21.15.50:6831/3135765006","heartbeat_front_addr":"172.21.15.50:6829/3135765006","state":["exists","up"]},{"osd":11,"uuid":"7b6e3d93-59e1-4853-b4a2-261b728c47a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6825","nonce":1482712612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6827","nonce":1482712612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6831","nonce":1482712612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6829","nonce":1482712612}]},"public_addr":"172.21.15.142:6825/1482712612","cluster_addr":"172.21.15.142:6827/1482712612","heartbeat_back_addr":"172.21.15.142:6831/1482712612","heartbeat_front_addr":"172.21.15.142:6829/1482712612","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:08.082015+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:32.634928+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:55.978864+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:21.796863+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:46.862518+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:11.440432+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:37.764777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:03.374958+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:28.714052+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:55.761167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:22.330951+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:47.839334+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/2465242088":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6801/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6800/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/1381997090":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/3507268217":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2570024356":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/2691061938":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2775954801":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6800/2896098005":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6801/2896098005":"2024-04-27T21:25:02.078730+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:34:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.709+0000 7ff5badf0640 1 -- 172.21.15.46:0/4256849860 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ff5a00a3c40 msgr2=0x7ff5a00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:37.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.709+0000 7ff5badf0640 1 --2- 172.21.15.46:0/4256849860 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ff5a00a3c40 0x7ff5a00a6100 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff5a4005c40 tx=0x7ff5a400a430 comp rx=0 tx=0).stop 2024-04-26T21:34:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.709+0000 7ff5badf0640 1 -- 172.21.15.46:0/4256849860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5b41033f0 msgr2=0x7ff5b4198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.709+0000 7ff5badf0640 1 --2- 172.21.15.46:0/4256849860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5b41033f0 0x7ff5b4198c60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff59c031500 tx=0x7ff59c0043c0 comp rx=0 tx=0).stop 2024-04-26T21:34:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.709+0000 7ff5badf0640 1 -- 172.21.15.46:0/4256849860 shutdown_connections 2024-04-26T21:34:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.709+0000 7ff5badf0640 1 --2- 172.21.15.46:0/4256849860 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ff5a00a3c40 0x7ff5a00a6100 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.709+0000 7ff5badf0640 1 --2- 172.21.15.46:0/4256849860 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ff5b4108650 0x7ff5b41a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.709+0000 7ff5badf0640 1 --2- 172.21.15.46:0/4256849860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5b4105d20 0x7ff5b41991a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.709+0000 7ff5badf0640 1 --2- 172.21.15.46:0/4256849860 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff5b41033f0 0x7ff5b4198c60 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.709+0000 7ff5badf0640 1 -- 172.21.15.46:0/4256849860 >> 172.21.15.46:0/4256849860 conn(0x7ff5b40fcde0 msgr2=0x7ff5b4103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.709+0000 7ff5badf0640 1 -- 172.21.15.46:0/4256849860 shutdown_connections 2024-04-26T21:34:37.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.709+0000 7ff5badf0640 1 -- 172.21.15.46:0/4256849860 wait complete. 2024-04-26T21:34:37.722 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T21:27:57.935988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T21:32:42.195211+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T21:32:43.162114+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T21:34:37.722 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T21:34:37.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.826+0000 7f5a717ec640 1 -- 172.21.15.46:0/1607988362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a6c102990 msgr2=0x7f5a6c102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:37.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.826+0000 7f5a717ec640 1 --2- 172.21.15.46:0/1607988362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a6c102990 0x7f5a6c102db0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c009900 tx=0x7f5a5c02a410 comp rx=0 tx=0).stop 2024-04-26T21:34:37.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.826+0000 7f5a717ec640 1 -- 172.21.15.46:0/1607988362 shutdown_connections 2024-04-26T21:34:37.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.826+0000 7f5a717ec640 1 --2- 172.21.15.46:0/1607988362 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a6c104540 0x7f5a6c10add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:37.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.826+0000 7f5a717ec640 1 --2- 172.21.15.46:0/1607988362 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a6c103b80 0x7f5a6c104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:37.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.826+0000 7f5a717ec640 1 --2- 172.21.15.46:0/1607988362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a6c102990 0x7f5a6c102db0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:37.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.826+0000 7f5a717ec640 1 -- 172.21.15.46:0/1607988362 >> 172.21.15.46:0/1607988362 conn(0x7f5a6c0fdf00 msgr2=0x7f5a6c100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:37.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.826+0000 7f5a717ec640 1 -- 172.21.15.46:0/1607988362 shutdown_connections 2024-04-26T21:34:37.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.826+0000 7f5a717ec640 1 -- 172.21.15.46:0/1607988362 wait complete. 2024-04-26T21:34:37.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.827+0000 7f5a717ec640 1 Processor -- start 2024-04-26T21:34:37.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.827+0000 7f5a717ec640 1 -- start start 2024-04-26T21:34:37.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.827+0000 7f5a717ec640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a6c102990 0x7f5a6c0766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:37.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.827+0000 7f5a717ec640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a6c103b80 0x7f5a6c074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:37.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.827+0000 7f5a717ec640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a6c104540 0x7f5a6c075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:37.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.827+0000 7f5a717ec640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6c10e580 con 0x7f5a6c102990 2024-04-26T21:34:37.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.827+0000 7f5a717ec640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6c10e400 con 0x7f5a6c104540 2024-04-26T21:34:37.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.827+0000 7f5a717ec640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6c10e700 con 0x7f5a6c103b80 2024-04-26T21:34:37.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.827+0000 7f5a6affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a6c102990 0x7f5a6c0766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:37.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.827+0000 7f5a6affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a6c102990 0x7f5a6c0766e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41820/0 (socket says 172.21.15.46:41820) 2024-04-26T21:34:37.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.827+0000 7f5a6affd640 1 -- 172.21.15.46:0/1750966675 learned_addr learned my addr 172.21.15.46:0/1750966675 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:37.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.827+0000 7f5a63fff640 1 --2- 172.21.15.46:0/1750966675 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a6c103b80 0x7f5a6c074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:37.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.827+0000 7f5a6affd640 1 -- 172.21.15.46:0/1750966675 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a6c103b80 msgr2=0x7f5a6c074d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:37.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.827+0000 7f5a6affd640 1 --2- 172.21.15.46:0/1750966675 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a6c103b80 0x7f5a6c074d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:37.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.827+0000 7f5a6affd640 1 -- 172.21.15.46:0/1750966675 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a6c104540 msgr2=0x7f5a6c075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:37.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.827+0000 7f5a6affd640 1 --2- 172.21.15.46:0/1750966675 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a6c104540 0x7f5a6c075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:37.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.827+0000 7f5a6affd640 1 -- 172.21.15.46:0/1750966675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a6c075af0 con 0x7f5a6c102990 2024-04-26T21:34:37.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.827+0000 7f5a63fff640 1 --2- 172.21.15.46:0/1750966675 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a6c103b80 0x7f5a6c074d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:34:37.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.827+0000 7f5a6affd640 1 --2- 172.21.15.46:0/1750966675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a6c102990 0x7f5a6c0766e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c031500 tx=0x7f5a5c003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:37.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.828+0000 7f5a68ff9640 1 -- 172.21.15.46:0/1750966675 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c03f070 con 0x7f5a6c102990 2024-04-26T21:34:37.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.828+0000 7f5a717ec640 1 -- 172.21.15.46:0/1750966675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a6c1a2d10 con 0x7f5a6c102990 2024-04-26T21:34:37.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.828+0000 7f5a68ff9640 1 -- 172.21.15.46:0/1750966675 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a5c003e70 con 0x7f5a6c102990 2024-04-26T21:34:37.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.828+0000 7f5a68ff9640 1 -- 172.21.15.46:0/1750966675 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c035030 con 0x7f5a6c102990 2024-04-26T21:34:37.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.828+0000 7f5a717ec640 1 -- 172.21.15.46:0/1750966675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a6c1a31a0 con 0x7f5a6c102990 2024-04-26T21:34:37.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.829+0000 7f5a717ec640 1 -- 172.21.15.46:0/1750966675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a6c109d50 con 0x7f5a6c102990 2024-04-26T21:34:37.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.830+0000 7f5a68ff9640 1 -- 172.21.15.46:0/1750966675 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5a5c042070 con 0x7f5a6c102990 2024-04-26T21:34:37.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.831+0000 7f5a68ff9640 1 --2- 172.21.15.46:0/1750966675 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5a580a3c40 0x7f5a580a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:37.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.831+0000 7f5a63fff640 1 --2- 172.21.15.46:0/1750966675 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5a580a3c40 0x7f5a580a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:37.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.831+0000 7f5a68ff9640 1 -- 172.21.15.46:0/1750966675 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f5a5c0f12f0 con 0x7f5a6c102990 2024-04-26T21:34:37.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.831+0000 7f5a63fff640 1 --2- 172.21.15.46:0/1750966675 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5a580a3c40 0x7f5a580a6100 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5a540058c0 tx=0x7f5a54005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:37.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:37.835+0000 7f5a68ff9640 1 -- 172.21.15.46:0/1750966675 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5a5c0bdb40 con 0x7f5a6c102990 2024-04-26T21:34:38.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.023+0000 7f5a717ec640 1 -- 172.21.15.46:0/1750966675 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5a6c02dd20 con 0x7f5a6c102990 2024-04-26T21:34:38.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.023+0000 7f5a68ff9640 1 -- 172.21.15.46:0/1750966675 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f5a5c0bd440 con 0x7f5a6c102990 2024-04-26T21:34:38.024 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-26T21:34:38.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.026+0000 7f5a717ec640 1 -- 172.21.15.46:0/1750966675 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5a580a3c40 msgr2=0x7f5a580a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:38.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.026+0000 7f5a717ec640 1 --2- 172.21.15.46:0/1750966675 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5a580a3c40 0x7f5a580a6100 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5a540058c0 tx=0x7f5a54005850 comp rx=0 tx=0).stop 2024-04-26T21:34:38.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.026+0000 7f5a717ec640 1 -- 172.21.15.46:0/1750966675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a6c102990 msgr2=0x7f5a6c0766e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:38.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.026+0000 7f5a717ec640 1 --2- 172.21.15.46:0/1750966675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a6c102990 0x7f5a6c0766e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c031500 tx=0x7f5a5c003c80 comp rx=0 tx=0).stop 2024-04-26T21:34:38.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.026+0000 7f5a717ec640 1 -- 172.21.15.46:0/1750966675 shutdown_connections 2024-04-26T21:34:38.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.026+0000 7f5a717ec640 1 --2- 172.21.15.46:0/1750966675 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f5a580a3c40 0x7f5a580a6100 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.026+0000 7f5a717ec640 1 --2- 172.21.15.46:0/1750966675 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a6c104540 0x7f5a6c075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.026+0000 7f5a717ec640 1 --2- 172.21.15.46:0/1750966675 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f5a6c103b80 0x7f5a6c074d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.026+0000 7f5a717ec640 1 --2- 172.21.15.46:0/1750966675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a6c102990 0x7f5a6c0766e0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.026+0000 7f5a717ec640 1 -- 172.21.15.46:0/1750966675 >> 172.21.15.46:0/1750966675 conn(0x7f5a6c0fdf00 msgr2=0x7f5a6c106f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:38.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.026+0000 7f5a717ec640 1 -- 172.21.15.46:0/1750966675 shutdown_connections 2024-04-26T21:34:38.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.026+0000 7f5a717ec640 1 -- 172.21.15.46:0/1750966675 wait complete. 2024-04-26T21:34:38.037 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:37 smithi046 ceph-mon[29201]: pgmap v350: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:34:38.038 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T21:34:38.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:37 smithi142 ceph-mon[32944]: pgmap v350: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:34:38.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:37 smithi050 ceph-mon[32753]: pgmap v350: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:34:38.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.142+0000 7f8b55190640 1 -- 172.21.15.46:0/1622887108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b50105d40 msgr2=0x7f8b50108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:38.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.142+0000 7f8b55190640 1 --2- 172.21.15.46:0/1622887108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b50105d40 0x7f8b50108130 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8b38009900 tx=0x7f8b3802a410 comp rx=0 tx=0).stop 2024-04-26T21:34:38.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.142+0000 7f8b55190640 1 -- 172.21.15.46:0/1622887108 shutdown_connections 2024-04-26T21:34:38.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.142+0000 7f8b55190640 1 --2- 172.21.15.46:0/1622887108 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b50108670 0x7f8b5010aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.142+0000 7f8b55190640 1 --2- 172.21.15.46:0/1622887108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b50105d40 0x7f8b50108130 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.142+0000 7f8b55190640 1 --2- 172.21.15.46:0/1622887108 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b50103410 0x7f8b50105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.142+0000 7f8b55190640 1 -- 172.21.15.46:0/1622887108 >> 172.21.15.46:0/1622887108 conn(0x7f8b500fce20 msgr2=0x7f8b500ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:38.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.142+0000 7f8b55190640 1 -- 172.21.15.46:0/1622887108 shutdown_connections 2024-04-26T21:34:38.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.142+0000 7f8b55190640 1 -- 172.21.15.46:0/1622887108 wait complete. 2024-04-26T21:34:38.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.143+0000 7f8b55190640 1 Processor -- start 2024-04-26T21:34:38.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.143+0000 7f8b55190640 1 -- start start 2024-04-26T21:34:38.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.143+0000 7f8b55190640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b50103410 0x7f8b50198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.143+0000 7f8b55190640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b50105d40 0x7f8b501991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.143+0000 7f8b55190640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b50108670 0x7f8b501a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.143+0000 7f8b55190640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8b5010e4b0 con 0x7f8b50108670 2024-04-26T21:34:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.143+0000 7f8b55190640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8b5010e330 con 0x7f8b50105d40 2024-04-26T21:34:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.143+0000 7f8b55190640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8b5010e630 con 0x7f8b50103410 2024-04-26T21:34:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.143+0000 7f8b4f59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b50108670 0x7f8b501a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.143+0000 7f8b4f59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b50108670 0x7f8b501a0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41834/0 (socket says 172.21.15.46:41834) 2024-04-26T21:34:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.143+0000 7f8b4f59e640 1 -- 172.21.15.46:0/3945503914 learned_addr learned my addr 172.21.15.46:0/3945503914 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.143+0000 7f8b47fff640 1 --2- 172.21.15.46:0/3945503914 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b50105d40 0x7f8b501991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.143+0000 7f8b4ed9d640 1 --2- 172.21.15.46:0/3945503914 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b50103410 0x7f8b50198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.144+0000 7f8b4f59e640 1 -- 172.21.15.46:0/3945503914 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b50103410 msgr2=0x7f8b50198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.144+0000 7f8b4f59e640 1 --2- 172.21.15.46:0/3945503914 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b50103410 0x7f8b50198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.144+0000 7f8b4f59e640 1 -- 172.21.15.46:0/3945503914 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b50105d40 msgr2=0x7f8b501991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.144+0000 7f8b4f59e640 1 --2- 172.21.15.46:0/3945503914 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b50105d40 0x7f8b501991b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.144+0000 7f8b4f59e640 1 -- 172.21.15.46:0/3945503914 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b501a0830 con 0x7f8b50108670 2024-04-26T21:34:38.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.144+0000 7f8b4f59e640 1 --2- 172.21.15.46:0/3945503914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b50108670 0x7f8b501a0190 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8b4800cad0 tx=0x7f8b4800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:38.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.144+0000 7f8b4cd99640 1 -- 172.21.15.46:0/3945503914 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b4800ae20 con 0x7f8b50108670 2024-04-26T21:34:38.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.144+0000 7f8b55190640 1 -- 172.21.15.46:0/3945503914 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b501a0b20 con 0x7f8b50108670 2024-04-26T21:34:38.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.144+0000 7f8b4cd99640 1 -- 172.21.15.46:0/3945503914 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b48005020 con 0x7f8b50108670 2024-04-26T21:34:38.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.144+0000 7f8b4cd99640 1 -- 172.21.15.46:0/3945503914 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b4800f6b0 con 0x7f8b50108670 2024-04-26T21:34:38.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.144+0000 7f8b55190640 1 -- 172.21.15.46:0/3945503914 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b501a1070 con 0x7f8b50108670 2024-04-26T21:34:38.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.145+0000 7f8b55190640 1 -- 172.21.15.46:0/3945503914 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b500fe910 con 0x7f8b50108670 2024-04-26T21:34:38.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.146+0000 7f8b4cd99640 1 -- 172.21.15.46:0/3945503914 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8b480051d0 con 0x7f8b50108670 2024-04-26T21:34:38.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.147+0000 7f8b4cd99640 1 --2- 172.21.15.46:0/3945503914 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8b3c0a3c40 0x7f8b3c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:38.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.147+0000 7f8b4ed9d640 1 --2- 172.21.15.46:0/3945503914 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8b3c0a3c40 0x7f8b3c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:38.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.147+0000 7f8b4cd99640 1 -- 172.21.15.46:0/3945503914 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f8b480d06f0 con 0x7f8b50108670 2024-04-26T21:34:38.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.147+0000 7f8b4ed9d640 1 --2- 172.21.15.46:0/3945503914 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8b3c0a3c40 0x7f8b3c0a6100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8b40005fd0 tx=0x7f8b40005e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:38.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.151+0000 7f8b4cd99640 1 -- 172.21.15.46:0/3945503914 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8b48014030 con 0x7f8b50108670 2024-04-26T21:34:38.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.336+0000 7f8b55190640 1 -- 172.21.15.46:0/3945503914 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8b5002dcc0 con 0x7f8b50108670 2024-04-26T21:34:38.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.337+0000 7f8b4cd99640 1 -- 172.21.15.46:0/3945503914 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f8b4809cf40 con 0x7f8b50108670 2024-04-26T21:34:38.338 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-26T21:34:38.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.338+0000 7f8b55190640 1 -- 172.21.15.46:0/3945503914 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8b3c0a3c40 msgr2=0x7f8b3c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:38.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.338+0000 7f8b55190640 1 --2- 172.21.15.46:0/3945503914 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8b3c0a3c40 0x7f8b3c0a6100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8b40005fd0 tx=0x7f8b40005e40 comp rx=0 tx=0).stop 2024-04-26T21:34:38.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.338+0000 7f8b55190640 1 -- 172.21.15.46:0/3945503914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b50108670 msgr2=0x7f8b501a0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:38.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.338+0000 7f8b55190640 1 --2- 172.21.15.46:0/3945503914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b50108670 0x7f8b501a0190 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8b4800cad0 tx=0x7f8b4800cf90 comp rx=0 tx=0).stop 2024-04-26T21:34:38.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.339+0000 7f8b55190640 1 -- 172.21.15.46:0/3945503914 shutdown_connections 2024-04-26T21:34:38.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.339+0000 7f8b55190640 1 --2- 172.21.15.46:0/3945503914 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8b3c0a3c40 0x7f8b3c0a6100 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.339+0000 7f8b55190640 1 --2- 172.21.15.46:0/3945503914 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b50108670 0x7f8b501a0190 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.339+0000 7f8b55190640 1 --2- 172.21.15.46:0/3945503914 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8b50105d40 0x7f8b501991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.339+0000 7f8b55190640 1 --2- 172.21.15.46:0/3945503914 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8b50103410 0x7f8b50198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.339+0000 7f8b55190640 1 -- 172.21.15.46:0/3945503914 >> 172.21.15.46:0/3945503914 conn(0x7f8b500fce20 msgr2=0x7f8b50103cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:38.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.339+0000 7f8b55190640 1 -- 172.21.15.46:0/3945503914 shutdown_connections 2024-04-26T21:34:38.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.339+0000 7f8b55190640 1 -- 172.21.15.46:0/3945503914 wait complete. 2024-04-26T21:34:38.351 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T21:34:38.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.455+0000 7fea6fdc9640 1 -- 172.21.15.46:0/4257061289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea68102990 msgr2=0x7fea68102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:38.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.455+0000 7fea6fdc9640 1 --2- 172.21.15.46:0/4257061289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea68102990 0x7fea68102db0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fea64009900 tx=0x7fea6402a410 comp rx=0 tx=0).stop 2024-04-26T21:34:38.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.455+0000 7fea6fdc9640 1 -- 172.21.15.46:0/4257061289 shutdown_connections 2024-04-26T21:34:38.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.455+0000 7fea6fdc9640 1 --2- 172.21.15.46:0/4257061289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea68104540 0x7fea6810add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.455+0000 7fea6fdc9640 1 --2- 172.21.15.46:0/4257061289 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea68103b80 0x7fea68104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.455+0000 7fea6fdc9640 1 --2- 172.21.15.46:0/4257061289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea68102990 0x7fea68102db0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.455+0000 7fea6fdc9640 1 -- 172.21.15.46:0/4257061289 >> 172.21.15.46:0/4257061289 conn(0x7fea680fdf00 msgr2=0x7fea68100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:38.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.455+0000 7fea6fdc9640 1 -- 172.21.15.46:0/4257061289 shutdown_connections 2024-04-26T21:34:38.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.455+0000 7fea6fdc9640 1 -- 172.21.15.46:0/4257061289 wait complete. 2024-04-26T21:34:38.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.456+0000 7fea6fdc9640 1 Processor -- start 2024-04-26T21:34:38.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.456+0000 7fea6fdc9640 1 -- start start 2024-04-26T21:34:38.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.456+0000 7fea6fdc9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea68102990 0x7fea680766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:38.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.456+0000 7fea6fdc9640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea68103b80 0x7fea68074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:38.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.456+0000 7fea6fdc9640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea68104540 0x7fea68075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.456+0000 7fea6fdc9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fea6810e490 con 0x7fea68102990 2024-04-26T21:34:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.456+0000 7fea6fdc9640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fea6810e310 con 0x7fea68104540 2024-04-26T21:34:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.456+0000 7fea6fdc9640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fea6810e610 con 0x7fea68103b80 2024-04-26T21:34:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.456+0000 7fea6db65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea68102990 0x7fea680766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.456+0000 7fea6db65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea68102990 0x7fea680766e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41864/0 (socket says 172.21.15.46:41864) 2024-04-26T21:34:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.456+0000 7fea6db65640 1 -- 172.21.15.46:0/1955374774 learned_addr learned my addr 172.21.15.46:0/1955374774 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.456+0000 7fea6db65640 1 -- 172.21.15.46:0/1955374774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea68103b80 msgr2=0x7fea68074d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.456+0000 7fea6db65640 1 --2- 172.21.15.46:0/1955374774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea68103b80 0x7fea68074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.456+0000 7fea6db65640 1 -- 172.21.15.46:0/1955374774 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea68104540 msgr2=0x7fea68075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.456+0000 7fea6db65640 1 --2- 172.21.15.46:0/1955374774 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea68104540 0x7fea68075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.456+0000 7fea6db65640 1 -- 172.21.15.46:0/1955374774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea68075af0 con 0x7fea68102990 2024-04-26T21:34:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.457+0000 7fea6db65640 1 --2- 172.21.15.46:0/1955374774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea68102990 0x7fea680766e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fea64006010 tx=0x7fea640042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.457+0000 7fea5effd640 1 -- 172.21.15.46:0/1955374774 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea6403f030 con 0x7fea68102990 2024-04-26T21:34:38.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.457+0000 7fea5effd640 1 -- 172.21.15.46:0/1955374774 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea64003cb0 con 0x7fea68102990 2024-04-26T21:34:38.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.457+0000 7fea6fdc9640 1 -- 172.21.15.46:0/1955374774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea681a2c30 con 0x7fea68102990 2024-04-26T21:34:38.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.457+0000 7fea5effd640 1 -- 172.21.15.46:0/1955374774 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea64035070 con 0x7fea68102990 2024-04-26T21:34:38.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.457+0000 7fea6fdc9640 1 -- 172.21.15.46:0/1955374774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea681a3180 con 0x7fea68102990 2024-04-26T21:34:38.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.458+0000 7fea6fdc9640 1 -- 172.21.15.46:0/1955374774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea68109d20 con 0x7fea68102990 2024-04-26T21:34:38.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.460+0000 7fea5effd640 1 -- 172.21.15.46:0/1955374774 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fea64042030 con 0x7fea68102990 2024-04-26T21:34:38.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.460+0000 7fea5effd640 1 --2- 172.21.15.46:0/1955374774 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fea540a3c40 0x7fea540a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:38.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.460+0000 7fea6d364640 1 --2- 172.21.15.46:0/1955374774 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fea540a3c40 0x7fea540a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:38.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.460+0000 7fea5effd640 1 -- 172.21.15.46:0/1955374774 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7fea640f0fe0 con 0x7fea68102990 2024-04-26T21:34:38.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.461+0000 7fea6d364640 1 --2- 172.21.15.46:0/1955374774 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fea540a3c40 0x7fea540a6100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fea68076450 tx=0x7fea58005ed0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:38.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.464+0000 7fea5effd640 1 -- 172.21.15.46:0/1955374774 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fea640bd890 con 0x7fea68102990 2024-04-26T21:34:38.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.653+0000 7fea6fdc9640 1 -- 172.21.15.46:0/1955374774 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fea6802dd60 con 0x7fea68102990 2024-04-26T21:34:38.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.653+0000 7fea5effd640 1 -- 172.21.15.46:0/1955374774 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7fea640bd190 con 0x7fea68102990 2024-04-26T21:34:38.654 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-26T21:34:38.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.655+0000 7fea6fdc9640 1 -- 172.21.15.46:0/1955374774 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fea540a3c40 msgr2=0x7fea540a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:38.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.655+0000 7fea6fdc9640 1 --2- 172.21.15.46:0/1955374774 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fea540a3c40 0x7fea540a6100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fea68076450 tx=0x7fea58005ed0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.655+0000 7fea6fdc9640 1 -- 172.21.15.46:0/1955374774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea68102990 msgr2=0x7fea680766e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:38.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.655+0000 7fea6fdc9640 1 --2- 172.21.15.46:0/1955374774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea68102990 0x7fea680766e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fea64006010 tx=0x7fea640042f0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.655+0000 7fea6fdc9640 1 -- 172.21.15.46:0/1955374774 shutdown_connections 2024-04-26T21:34:38.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.655+0000 7fea6fdc9640 1 --2- 172.21.15.46:0/1955374774 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fea540a3c40 0x7fea540a6100 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.655+0000 7fea6fdc9640 1 --2- 172.21.15.46:0/1955374774 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea68104540 0x7fea68075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.655+0000 7fea6fdc9640 1 --2- 172.21.15.46:0/1955374774 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fea68103b80 0x7fea68074d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.655+0000 7fea6fdc9640 1 --2- 172.21.15.46:0/1955374774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea68102990 0x7fea680766e0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.655+0000 7fea6fdc9640 1 -- 172.21.15.46:0/1955374774 >> 172.21.15.46:0/1955374774 conn(0x7fea680fdf00 msgr2=0x7fea68106ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:38.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.655+0000 7fea6fdc9640 1 -- 172.21.15.46:0/1955374774 shutdown_connections 2024-04-26T21:34:38.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.655+0000 7fea6fdc9640 1 -- 172.21.15.46:0/1955374774 wait complete. 2024-04-26T21:34:38.667 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:34:38.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.768+0000 7f05d612b640 1 -- 172.21.15.46:0/445824981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05d0103b70 msgr2=0x7f05d0103ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:38.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.768+0000 7f05d612b640 1 --2- 172.21.15.46:0/445824981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05d0103b70 0x7f05d0103ff0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f05c4009900 tx=0x7f05c402a410 comp rx=0 tx=0).stop 2024-04-26T21:34:38.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.768+0000 7f05d612b640 1 -- 172.21.15.46:0/445824981 shutdown_connections 2024-04-26T21:34:38.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.768+0000 7f05d612b640 1 --2- 172.21.15.46:0/445824981 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05d0104530 0x7f05d010adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.768+0000 7f05d612b640 1 --2- 172.21.15.46:0/445824981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05d0103b70 0x7f05d0103ff0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.768+0000 7f05d612b640 1 --2- 172.21.15.46:0/445824981 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f05d0102980 0x7f05d0102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.768+0000 7f05d612b640 1 -- 172.21.15.46:0/445824981 >> 172.21.15.46:0/445824981 conn(0x7f05d00fdf10 msgr2=0x7f05d0100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:38.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.768+0000 7f05d612b640 1 -- 172.21.15.46:0/445824981 shutdown_connections 2024-04-26T21:34:38.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.768+0000 7f05d612b640 1 -- 172.21.15.46:0/445824981 wait complete. 2024-04-26T21:34:38.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.769+0000 7f05d612b640 1 Processor -- start 2024-04-26T21:34:38.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.769+0000 7f05d612b640 1 -- start start 2024-04-26T21:34:38.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.769+0000 7f05d612b640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05d0102980 0x7f05d019ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:38.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.769+0000 7f05d612b640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f05d0103b70 0x7f05d019b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:38.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.769+0000 7f05d612b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05d0104530 0x7f05d01a23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:38.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.769+0000 7f05d612b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f05d010e4b0 con 0x7f05d0104530 2024-04-26T21:34:38.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.769+0000 7f05d612b640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f05d010e330 con 0x7f05d0102980 2024-04-26T21:34:38.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.769+0000 7f05d612b640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f05d010e630 con 0x7f05d0103b70 2024-04-26T21:34:38.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.769+0000 7f05cffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05d0104530 0x7f05d01a23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:38.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.769+0000 7f05cffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05d0104530 0x7f05d01a23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41884/0 (socket says 172.21.15.46:41884) 2024-04-26T21:34:38.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.769+0000 7f05cffff640 1 -- 172.21.15.46:0/3983681706 learned_addr learned my addr 172.21.15.46:0/3983681706 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:38.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.769+0000 7f05cf7fe640 1 --2- 172.21.15.46:0/3983681706 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05d0102980 0x7f05d019ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:38.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.769+0000 7f05ceffd640 1 --2- 172.21.15.46:0/3983681706 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f05d0103b70 0x7f05d019b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:38.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.770+0000 7f05cffff640 1 -- 172.21.15.46:0/3983681706 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f05d0103b70 msgr2=0x7f05d019b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:38.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.770+0000 7f05cffff640 1 --2- 172.21.15.46:0/3983681706 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f05d0103b70 0x7f05d019b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.770+0000 7f05cffff640 1 -- 172.21.15.46:0/3983681706 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05d0102980 msgr2=0x7f05d019ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:38.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.770+0000 7f05cffff640 1 --2- 172.21.15.46:0/3983681706 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05d0102980 0x7f05d019ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:38.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.770+0000 7f05cffff640 1 -- 172.21.15.46:0/3983681706 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05d01a2a40 con 0x7f05d0104530 2024-04-26T21:34:38.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.770+0000 7f05ceffd640 1 --2- 172.21.15.46:0/3983681706 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f05d0103b70 0x7f05d019b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:34:38.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.770+0000 7f05cffff640 1 --2- 172.21.15.46:0/3983681706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05d0104530 0x7f05d01a23d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f05c800eaa0 tx=0x7f05c800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:38.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.770+0000 7f05ccff9640 1 -- 172.21.15.46:0/3983681706 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c800cdb0 con 0x7f05d0104530 2024-04-26T21:34:38.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.770+0000 7f05d612b640 1 -- 172.21.15.46:0/3983681706 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05d01a2d30 con 0x7f05d0104530 2024-04-26T21:34:38.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.770+0000 7f05ccff9640 1 -- 172.21.15.46:0/3983681706 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05c8005020 con 0x7f05d0104530 2024-04-26T21:34:38.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.770+0000 7f05ccff9640 1 -- 172.21.15.46:0/3983681706 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c80116d0 con 0x7f05d0104530 2024-04-26T21:34:38.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.770+0000 7f05d612b640 1 -- 172.21.15.46:0/3983681706 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05d01a31d0 con 0x7f05d0104530 2024-04-26T21:34:38.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.771+0000 7f05d612b640 1 -- 172.21.15.46:0/3983681706 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05d01a3460 con 0x7f05d0104530 2024-04-26T21:34:38.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.772+0000 7f05ccff9640 1 -- 172.21.15.46:0/3983681706 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f05c80051d0 con 0x7f05d0104530 2024-04-26T21:34:38.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.773+0000 7f05ccff9640 1 --2- 172.21.15.46:0/3983681706 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f05b80a3c40 0x7f05b80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:38.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.773+0000 7f05cf7fe640 1 --2- 172.21.15.46:0/3983681706 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f05b80a3c40 0x7f05b80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:38.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.773+0000 7f05ccff9640 1 -- 172.21.15.46:0/3983681706 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f05c80d0830 con 0x7f05d0104530 2024-04-26T21:34:38.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.773+0000 7f05cf7fe640 1 --2- 172.21.15.46:0/3983681706 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f05b80a3c40 0x7f05b80a6100 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f05c0005fd0 tx=0x7f05c0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:38.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:38.777+0000 7f05ccff9640 1 -- 172.21.15.46:0/3983681706 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f05c8016030 con 0x7f05d0104530 2024-04-26T21:34:39.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.003+0000 7f05d612b640 1 -- 172.21.15.46:0/3983681706 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05d01a37c0 con 0x7f05d0104530 2024-04-26T21:34:39.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.004+0000 7f05ccff9640 1 -- 172.21.15.46:0/3983681706 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11863 (secure 0 0 0) 0x7f05c809d080 con 0x7f05d0104530 2024-04-26T21:34:39.005 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:34:39.005 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3674296286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3674296286},{"type":"v1","addr":"172.21.15.46:6835","nonce":3674296286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/391791206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":391791206},{"type":"v1","addr":"172.21.15.46:6837","nonce":391791206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/1222489912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1222489912},{"type":"v1","addr":"172.21.15.46:6839","nonce":1222489912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14658,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/860605548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":860605548},{"type":"v1","addr":"172.21.15.46:6841","nonce":860605548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3621399933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3621399933},{"type":"v1","addr":"172.21.15.142:6833","nonce":3621399933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/345015778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":345015778},{"type":"v1","addr":"172.21.15.50:6833","nonce":345015778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T21:32:44.201541+0000","modified":"2024-04-26T21:32:49.889029+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24526,"mds_2":24509},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/2546509863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":2546509863},{"type":"v1","addr":"172.21.15.50:6835","nonce":2546509863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/3791468066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":3791468066},{"type":"v1","addr":"172.21.15.142:6835","nonce":3791468066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6837/1723132553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1723132553},{"type":"v1","addr":"172.21.15.142:6837","nonce":1723132553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6837/3189620395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3189620395},{"type":"v1","addr":"172.21.15.50:6837","nonce":3189620395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24526":{"gid":24526,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.142:6839/3089509101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3089509101},{"type":"v1","addr":"172.21.15.142:6839","nonce":3089509101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6839/2453253368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2453253368},{"type":"v1","addr":"172.21.15.50:6839","nonce":2453253368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24509,24526,24539]},"id":1}]} 2024-04-26T21:34:39.005 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 22 2024-04-26T21:34:39.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.006+0000 7f05d612b640 1 -- 172.21.15.46:0/3983681706 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f05b80a3c40 msgr2=0x7f05b80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:39.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.006+0000 7f05d612b640 1 --2- 172.21.15.46:0/3983681706 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f05b80a3c40 0x7f05b80a6100 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f05c0005fd0 tx=0x7f05c0005850 comp rx=0 tx=0).stop 2024-04-26T21:34:39.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.006+0000 7f05d612b640 1 -- 172.21.15.46:0/3983681706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05d0104530 msgr2=0x7f05d01a23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:39.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.006+0000 7f05d612b640 1 --2- 172.21.15.46:0/3983681706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05d0104530 0x7f05d01a23d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f05c800eaa0 tx=0x7f05c800ef60 comp rx=0 tx=0).stop 2024-04-26T21:34:39.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.006+0000 7f05d612b640 1 -- 172.21.15.46:0/3983681706 shutdown_connections 2024-04-26T21:34:39.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.006+0000 7f05d612b640 1 --2- 172.21.15.46:0/3983681706 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f05b80a3c40 0x7f05b80a6100 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.006+0000 7f05d612b640 1 --2- 172.21.15.46:0/3983681706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05d0104530 0x7f05d01a23d0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.006+0000 7f05d612b640 1 --2- 172.21.15.46:0/3983681706 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f05d0103b70 0x7f05d019b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.006+0000 7f05d612b640 1 --2- 172.21.15.46:0/3983681706 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f05d0102980 0x7f05d019ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.006+0000 7f05d612b640 1 -- 172.21.15.46:0/3983681706 >> 172.21.15.46:0/3983681706 conn(0x7f05d00fdf10 msgr2=0x7f05d0074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:39.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.006+0000 7f05d612b640 1 -- 172.21.15.46:0/3983681706 shutdown_connections 2024-04-26T21:34:39.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.006+0000 7f05d612b640 1 -- 172.21.15.46:0/3983681706 wait complete. 2024-04-26T21:34:39.017 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:38 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/4256849860' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:34:39.017 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:38 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1750966675' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:34:39.017 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:38 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3945503914' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:34:39.019 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:34:39.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:38 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/4256849860' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:34:39.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:38 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/1750966675' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:34:39.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:38 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3945503914' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:34:39.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:38 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/4256849860' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:34:39.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:38 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/1750966675' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:34:39.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:38 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3945503914' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:34:39.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.119+0000 7fe8537af640 1 -- 172.21.15.46:0/4265909856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe84c102a80 msgr2=0x7fe84c102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:39.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.119+0000 7fe8537af640 1 --2- 172.21.15.46:0/4265909856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe84c102a80 0x7fe84c102f00 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fe83c009900 tx=0x7fe83c02a410 comp rx=0 tx=0).stop 2024-04-26T21:34:39.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.119+0000 7fe8537af640 1 -- 172.21.15.46:0/4265909856 shutdown_connections 2024-04-26T21:34:39.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.119+0000 7fe8537af640 1 --2- 172.21.15.46:0/4265909856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe84c103440 0x7fe84c109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.119+0000 7fe8537af640 1 --2- 172.21.15.46:0/4265909856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe84c102a80 0x7fe84c102f00 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.119+0000 7fe8537af640 1 --2- 172.21.15.46:0/4265909856 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe84c101890 0x7fe84c101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.119+0000 7fe8537af640 1 -- 172.21.15.46:0/4265909856 >> 172.21.15.46:0/4265909856 conn(0x7fe84c0fce20 msgr2=0x7fe84c0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:39.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.119+0000 7fe8537af640 1 -- 172.21.15.46:0/4265909856 shutdown_connections 2024-04-26T21:34:39.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.119+0000 7fe8537af640 1 -- 172.21.15.46:0/4265909856 wait complete. 2024-04-26T21:34:39.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.120+0000 7fe8537af640 1 Processor -- start 2024-04-26T21:34:39.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.120+0000 7fe8537af640 1 -- start start 2024-04-26T21:34:39.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.120+0000 7fe8537af640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe84c101890 0x7fe84c198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:39.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.120+0000 7fe8537af640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe84c102a80 0x7fe84c199140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:39.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.120+0000 7fe8537af640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe84c103440 0x7fe84c1a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:39.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.120+0000 7fe8537af640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe84c10e540 con 0x7fe84c103440 2024-04-26T21:34:39.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.120+0000 7fe8537af640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe84c10e3c0 con 0x7fe84c101890 2024-04-26T21:34:39.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.120+0000 7fe8537af640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe84c10e6c0 con 0x7fe84c102a80 2024-04-26T21:34:39.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.120+0000 7fe851d4c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe84c103440 0x7fe84c1a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:39.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.120+0000 7fe851d4c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe84c103440 0x7fe84c1a0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41894/0 (socket says 172.21.15.46:41894) 2024-04-26T21:34:39.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.120+0000 7fe851d4c640 1 -- 172.21.15.46:0/1951226289 learned_addr learned my addr 172.21.15.46:0/1951226289 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:39.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.120+0000 7fe85154b640 1 --2- 172.21.15.46:0/1951226289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe84c101890 0x7fe84c198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:39.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.120+0000 7fe850d4a640 1 --2- 172.21.15.46:0/1951226289 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe84c102a80 0x7fe84c199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:39.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.120+0000 7fe851d4c640 1 -- 172.21.15.46:0/1951226289 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe84c102a80 msgr2=0x7fe84c199140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:39.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.120+0000 7fe851d4c640 1 --2- 172.21.15.46:0/1951226289 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe84c102a80 0x7fe84c199140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.120+0000 7fe851d4c640 1 -- 172.21.15.46:0/1951226289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe84c101890 msgr2=0x7fe84c198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:39.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.120+0000 7fe851d4c640 1 --2- 172.21.15.46:0/1951226289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe84c101890 0x7fe84c198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.120+0000 7fe851d4c640 1 -- 172.21.15.46:0/1951226289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe84c1a07d0 con 0x7fe84c103440 2024-04-26T21:34:39.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.120+0000 7fe85154b640 1 --2- 172.21.15.46:0/1951226289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe84c101890 0x7fe84c198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:34:39.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.121+0000 7fe851d4c640 1 --2- 172.21.15.46:0/1951226289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe84c103440 0x7fe84c1a0160 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe84400eb10 tx=0x7fe84400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:39.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.121+0000 7fe8427fc640 1 -- 172.21.15.46:0/1951226289 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe844004b20 con 0x7fe84c103440 2024-04-26T21:34:39.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.121+0000 7fe8427fc640 1 -- 172.21.15.46:0/1951226289 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe844004cc0 con 0x7fe84c103440 2024-04-26T21:34:39.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.121+0000 7fe8427fc640 1 -- 172.21.15.46:0/1951226289 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe844010670 con 0x7fe84c103440 2024-04-26T21:34:39.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.121+0000 7fe8537af640 1 -- 172.21.15.46:0/1951226289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe84c1a0ac0 con 0x7fe84c103440 2024-04-26T21:34:39.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.121+0000 7fe8537af640 1 -- 172.21.15.46:0/1951226289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe84c1a0fc0 con 0x7fe84c103440 2024-04-26T21:34:39.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.122+0000 7fe8537af640 1 -- 172.21.15.46:0/1951226289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe84c108c80 con 0x7fe84c103440 2024-04-26T21:34:39.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.123+0000 7fe8427fc640 1 -- 172.21.15.46:0/1951226289 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe844005020 con 0x7fe84c103440 2024-04-26T21:34:39.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.123+0000 7fe8427fc640 1 --2- 172.21.15.46:0/1951226289 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe8380a3c40 0x7fe8380a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:39.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.124+0000 7fe85154b640 1 --2- 172.21.15.46:0/1951226289 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe8380a3c40 0x7fe8380a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:39.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.124+0000 7fe8427fc640 1 -- 172.21.15.46:0/1951226289 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7fe8440d0ca0 con 0x7fe84c103440 2024-04-26T21:34:39.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.124+0000 7fe85154b640 1 --2- 172.21.15.46:0/1951226289 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe8380a3c40 0x7fe8380a6100 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe84c1028e0 tx=0x7fe848005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:39.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.128+0000 7fe8427fc640 1 -- 172.21.15.46:0/1951226289 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe84409d4f0 con 0x7fe84c103440 2024-04-26T21:34:39.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.314+0000 7fe8537af640 1 -- 172.21.15.46:0/1951226289 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe84c02dd20 con 0x7fe84c103440 2024-04-26T21:34:39.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.316+0000 7fe8427fc640 1 -- 172.21.15.46:0/1951226289 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22602 (secure 0 0 0) 0x7fe84409cdf0 con 0x7fe84c103440 2024-04-26T21:34:39.317 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:34:39.317 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":87,"fsid":"36c7013a-0413-11ef-bc93-c7b262605968","created":"2024-04-26T21:24:30.481156+0000","modified":"2024-04-26T21:33:31.051554+0000","last_up_change":"2024-04-26T21:31:50.100491+0000","last_in_change":"2024-04-26T21:31:29.643428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:27:57.935988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:32:42.195211+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:32:43.162114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f53c8702-e4da-435f-a9dc-3911d0d36ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6803","nonce":2762665125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6805","nonce":2762665125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6809","nonce":2762665125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6807","nonce":2762665125}]},"public_addr":"172.21.15.46:6803/2762665125","cluster_addr":"172.21.15.46:6805/2762665125","heartbeat_back_addr":"172.21.15.46:6809/2762665125","heartbeat_front_addr":"172.21.15.46:6807/2762665125","state":["exists","up"]},{"osd":1,"uuid":"fc4d6f52-fc8a-4efd-80c9-c092b57e7f42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6801","nonce":1628008970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6803","nonce":1628008970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6807","nonce":1628008970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6805","nonce":1628008970}]},"public_addr":"172.21.15.50:6801/1628008970","cluster_addr":"172.21.15.50:6803/1628008970","heartbeat_back_addr":"172.21.15.50:6807/1628008970","heartbeat_front_addr":"172.21.15.50:6805/1628008970","state":["exists","up"]},{"osd":2,"uuid":"26d1baa1-4ea1-4b7e-9fcf-4f8bbf7329c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6801","nonce":1513279137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6803","nonce":1513279137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6807","nonce":1513279137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6805","nonce":1513279137}]},"public_addr":"172.21.15.142:6801/1513279137","cluster_addr":"172.21.15.142:6803/1513279137","heartbeat_back_addr":"172.21.15.142:6807/1513279137","heartbeat_front_addr":"172.21.15.142:6805/1513279137","state":["exists","up"]},{"osd":3,"uuid":"34d2401f-5fcc-491c-8fcf-c3c492611f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6811","nonce":2435181760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6813","nonce":2435181760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6817","nonce":2435181760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6815","nonce":2435181760}]},"public_addr":"172.21.15.46:6811/2435181760","cluster_addr":"172.21.15.46:6813/2435181760","heartbeat_back_addr":"172.21.15.46:6817/2435181760","heartbeat_front_addr":"172.21.15.46:6815/2435181760","state":["exists","up"]},{"osd":4,"uuid":"986bb709-e4cf-400d-982e-c3f5ac413e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6809","nonce":3177333250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6811","nonce":3177333250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6815","nonce":3177333250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6813","nonce":3177333250}]},"public_addr":"172.21.15.50:6809/3177333250","cluster_addr":"172.21.15.50:6811/3177333250","heartbeat_back_addr":"172.21.15.50:6815/3177333250","heartbeat_front_addr":"172.21.15.50:6813/3177333250","state":["exists","up"]},{"osd":5,"uuid":"b7d65851-ebe1-45e2-833d-8022446f2698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6809","nonce":4113107737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6811","nonce":4113107737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6815","nonce":4113107737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6813","nonce":4113107737}]},"public_addr":"172.21.15.142:6809/4113107737","cluster_addr":"172.21.15.142:6811/4113107737","heartbeat_back_addr":"172.21.15.142:6815/4113107737","heartbeat_front_addr":"172.21.15.142:6813/4113107737","state":["exists","up"]},{"osd":6,"uuid":"2a3e96d7-d5bf-44ff-9134-29f1420c0e4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6819","nonce":2325162882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6821","nonce":2325162882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6825","nonce":2325162882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6823","nonce":2325162882}]},"public_addr":"172.21.15.46:6819/2325162882","cluster_addr":"172.21.15.46:6821/2325162882","heartbeat_back_addr":"172.21.15.46:6825/2325162882","heartbeat_front_addr":"172.21.15.46:6823/2325162882","state":["exists","up"]},{"osd":7,"uuid":"e53ae885-23b6-46fd-b4b1-c62f31a6921f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6817","nonce":820310228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6819","nonce":820310228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6823","nonce":820310228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6821","nonce":820310228}]},"public_addr":"172.21.15.50:6817/820310228","cluster_addr":"172.21.15.50:6819/820310228","heartbeat_back_addr":"172.21.15.50:6823/820310228","heartbeat_front_addr":"172.21.15.50:6821/820310228","state":["exists","up"]},{"osd":8,"uuid":"356e2692-16a9-4045-b58f-c8f9cb5d7241","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6817","nonce":1824918711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6819","nonce":1824918711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6823","nonce":1824918711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6821","nonce":1824918711}]},"public_addr":"172.21.15.142:6817/1824918711","cluster_addr":"172.21.15.142:6819/1824918711","heartbeat_back_addr":"172.21.15.142:6823/1824918711","heartbeat_front_addr":"172.21.15.142:6821/1824918711","state":["exists","up"]},{"osd":9,"uuid":"8e55570e-77e5-4892-8b9f-77dcd03a4eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6827","nonce":3601947736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6829","nonce":3601947736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6833","nonce":3601947736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6831","nonce":3601947736}]},"public_addr":"172.21.15.46:6827/3601947736","cluster_addr":"172.21.15.46:6829/3601947736","heartbeat_back_addr":"172.21.15.46:6833/3601947736","heartbeat_front_addr":"172.21.15.46:6831/3601947736","state":["exists","up"]},{"osd":10,"uuid":"366389fa-fb9a-408b-b5c2-343c054c5a64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6825","nonce":3135765006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6827","nonce":3135765006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6831","nonce":3135765006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6829","nonce":3135765006}]},"public_addr":"172.21.15.50:6825/3135765006","cluster_addr":"172.21.15.50:6827/3135765006","heartbeat_back_addr":"172.21.15.50:6831/3135765006","heartbeat_front_addr":"172.21.15.50:6829/3135765006","state":["exists","up"]},{"osd":11,"uuid":"7b6e3d93-59e1-4853-b4a2-261b728c47a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6825","nonce":1482712612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6827","nonce":1482712612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6831","nonce":1482712612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6829","nonce":1482712612}]},"public_addr":"172.21.15.142:6825/1482712612","cluster_addr":"172.21.15.142:6827/1482712612","heartbeat_back_addr":"172.21.15.142:6831/1482712612","heartbeat_front_addr":"172.21.15.142:6829/1482712612","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:08.082015+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:32.634928+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:55.978864+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:21.796863+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:46.862518+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:11.440432+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:37.764777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:03.374958+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:28.714052+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:55.761167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:22.330951+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:47.839334+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/2465242088":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6801/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6800/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/1381997090":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/3507268217":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2570024356":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/2691061938":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2775954801":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6800/2896098005":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6801/2896098005":"2024-04-27T21:25:02.078730+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:34:39.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.317+0000 7fe8537af640 1 -- 172.21.15.46:0/1951226289 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe8380a3c40 msgr2=0x7fe8380a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:39.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.317+0000 7fe8537af640 1 --2- 172.21.15.46:0/1951226289 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe8380a3c40 0x7fe8380a6100 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe84c1028e0 tx=0x7fe848005f90 comp rx=0 tx=0).stop 2024-04-26T21:34:39.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.317+0000 7fe8537af640 1 -- 172.21.15.46:0/1951226289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe84c103440 msgr2=0x7fe84c1a0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:39.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.317+0000 7fe8537af640 1 --2- 172.21.15.46:0/1951226289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe84c103440 0x7fe84c1a0160 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe84400eb10 tx=0x7fe84400c2d0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.317+0000 7fe8537af640 1 -- 172.21.15.46:0/1951226289 shutdown_connections 2024-04-26T21:34:39.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.317+0000 7fe8537af640 1 --2- 172.21.15.46:0/1951226289 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe8380a3c40 0x7fe8380a6100 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.317+0000 7fe8537af640 1 --2- 172.21.15.46:0/1951226289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe84c103440 0x7fe84c1a0160 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.318+0000 7fe8537af640 1 --2- 172.21.15.46:0/1951226289 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe84c102a80 0x7fe84c199140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.318+0000 7fe8537af640 1 --2- 172.21.15.46:0/1951226289 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe84c101890 0x7fe84c198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.318+0000 7fe8537af640 1 -- 172.21.15.46:0/1951226289 >> 172.21.15.46:0/1951226289 conn(0x7fe84c0fce20 msgr2=0x7fe84c105e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:39.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.318+0000 7fe8537af640 1 -- 172.21.15.46:0/1951226289 shutdown_connections 2024-04-26T21:34:39.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.318+0000 7fe8537af640 1 -- 172.21.15.46:0/1951226289 wait complete. 2024-04-26T21:34:39.330 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T21:27:57.935988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T21:32:42.195211+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T21:32:43.162114+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T21:34:39.330 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T21:34:39.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.433+0000 7f6ec056a640 1 -- 172.21.15.46:0/1554631341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6eb8102a60 msgr2=0x7f6eb8102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:39.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.433+0000 7f6ec056a640 1 --2- 172.21.15.46:0/1554631341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6eb8102a60 0x7f6eb8102ee0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f6eac009900 tx=0x7f6eac02a410 comp rx=0 tx=0).stop 2024-04-26T21:34:39.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.433+0000 7f6ec056a640 1 -- 172.21.15.46:0/1554631341 shutdown_connections 2024-04-26T21:34:39.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.433+0000 7f6ec056a640 1 --2- 172.21.15.46:0/1554631341 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6eb8103420 0x7f6eb8109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.433+0000 7f6ec056a640 1 --2- 172.21.15.46:0/1554631341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6eb8102a60 0x7f6eb8102ee0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.433+0000 7f6ec056a640 1 --2- 172.21.15.46:0/1554631341 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6eb8101870 0x7f6eb8101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.433+0000 7f6ec056a640 1 -- 172.21.15.46:0/1554631341 >> 172.21.15.46:0/1554631341 conn(0x7f6eb80fcde0 msgr2=0x7f6eb80ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:39.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.434+0000 7f6ec056a640 1 -- 172.21.15.46:0/1554631341 shutdown_connections 2024-04-26T21:34:39.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.434+0000 7f6ec056a640 1 -- 172.21.15.46:0/1554631341 wait complete. 2024-04-26T21:34:39.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.434+0000 7f6ec056a640 1 Processor -- start 2024-04-26T21:34:39.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.434+0000 7f6ec056a640 1 -- start start 2024-04-26T21:34:39.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.434+0000 7f6ec056a640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6eb8101870 0x7f6eb8198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:39.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.434+0000 7f6ec056a640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6eb8102a60 0x7f6eb8199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:39.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.434+0000 7f6ec056a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6eb8103420 0x7f6eb81a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:39.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.434+0000 7f6ec056a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb810d3a0 con 0x7f6eb8103420 2024-04-26T21:34:39.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.434+0000 7f6ec056a640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb810d220 con 0x7f6eb8102a60 2024-04-26T21:34:39.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.434+0000 7f6ec056a640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb810d520 con 0x7f6eb8101870 2024-04-26T21:34:39.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.434+0000 7f6ebeb07640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6eb8103420 0x7f6eb81a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:39.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.434+0000 7f6ebeb07640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6eb8103420 0x7f6eb81a01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41914/0 (socket says 172.21.15.46:41914) 2024-04-26T21:34:39.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.434+0000 7f6ebeb07640 1 -- 172.21.15.46:0/46309026 learned_addr learned my addr 172.21.15.46:0/46309026 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:39.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.434+0000 7f6ebdb05640 1 --2- 172.21.15.46:0/46309026 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6eb8102a60 0x7f6eb8199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:39.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.434+0000 7f6ebe306640 1 --2- 172.21.15.46:0/46309026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6eb8101870 0x7f6eb8198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:39.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.435+0000 7f6ebeb07640 1 -- 172.21.15.46:0/46309026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6eb8101870 msgr2=0x7f6eb8198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:39.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.435+0000 7f6ebeb07640 1 --2- 172.21.15.46:0/46309026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6eb8101870 0x7f6eb8198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.435+0000 7f6ebeb07640 1 -- 172.21.15.46:0/46309026 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6eb8102a60 msgr2=0x7f6eb8199170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:39.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.435+0000 7f6ebeb07640 1 --2- 172.21.15.46:0/46309026 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6eb8102a60 0x7f6eb8199170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.435+0000 7f6ebeb07640 1 -- 172.21.15.46:0/46309026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eb81a0840 con 0x7f6eb8103420 2024-04-26T21:34:39.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.435+0000 7f6ebeb07640 1 --2- 172.21.15.46:0/46309026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6eb8103420 0x7f6eb81a01a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f6ea800e970 tx=0x7f6ea800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:39.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.435+0000 7f6eb77fe640 1 -- 172.21.15.46:0/46309026 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea800ccc0 con 0x7f6eb8103420 2024-04-26T21:34:39.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.435+0000 7f6eb77fe640 1 -- 172.21.15.46:0/46309026 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ea800ce60 con 0x7f6eb8103420 2024-04-26T21:34:39.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.435+0000 7f6ec056a640 1 -- 172.21.15.46:0/46309026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eb81a0b30 con 0x7f6eb8103420 2024-04-26T21:34:39.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.435+0000 7f6eb77fe640 1 -- 172.21.15.46:0/46309026 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea8010430 con 0x7f6eb8103420 2024-04-26T21:34:39.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.435+0000 7f6ec056a640 1 -- 172.21.15.46:0/46309026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eb81a1030 con 0x7f6eb8103420 2024-04-26T21:34:39.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.436+0000 7f6ec056a640 1 -- 172.21.15.46:0/46309026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6eb81a1200 con 0x7f6eb8103420 2024-04-26T21:34:39.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.437+0000 7f6eb77fe640 1 -- 172.21.15.46:0/46309026 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6ea8004ff0 con 0x7f6eb8103420 2024-04-26T21:34:39.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.438+0000 7f6eb77fe640 1 --2- 172.21.15.46:0/46309026 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6ea00a3c40 0x7f6ea00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:39.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.438+0000 7f6ebe306640 1 --2- 172.21.15.46:0/46309026 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6ea00a3c40 0x7f6ea00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:39.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.438+0000 7f6eb77fe640 1 -- 172.21.15.46:0/46309026 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f6ea80d0a40 con 0x7f6eb8103420 2024-04-26T21:34:39.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.438+0000 7f6ebe306640 1 --2- 172.21.15.46:0/46309026 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6ea00a3c40 0x7f6ea00a6100 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6eb819a100 tx=0x7f6eb0009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:39.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.443+0000 7f6eb77fe640 1 -- 172.21.15.46:0/46309026 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6ea809d290 con 0x7f6eb8103420 2024-04-26T21:34:39.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.627+0000 7f6ec056a640 1 -- 172.21.15.46:0/46309026 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6eb81a13d0 con 0x7f6eb8103420 2024-04-26T21:34:39.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.628+0000 7f6eb77fe640 1 -- 172.21.15.46:0/46309026 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f6ea809cb90 con 0x7f6eb8103420 2024-04-26T21:34:39.629 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-26T21:34:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.630+0000 7f6ec056a640 1 -- 172.21.15.46:0/46309026 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6ea00a3c40 msgr2=0x7f6ea00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.630+0000 7f6ec056a640 1 --2- 172.21.15.46:0/46309026 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6ea00a3c40 0x7f6ea00a6100 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6eb819a100 tx=0x7f6eb0009680 comp rx=0 tx=0).stop 2024-04-26T21:34:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.630+0000 7f6ec056a640 1 -- 172.21.15.46:0/46309026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6eb8103420 msgr2=0x7f6eb81a01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.630+0000 7f6ec056a640 1 --2- 172.21.15.46:0/46309026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6eb8103420 0x7f6eb81a01a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f6ea800e970 tx=0x7f6ea800ed70 comp rx=0 tx=0).stop 2024-04-26T21:34:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.630+0000 7f6ec056a640 1 -- 172.21.15.46:0/46309026 shutdown_connections 2024-04-26T21:34:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.630+0000 7f6ec056a640 1 --2- 172.21.15.46:0/46309026 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6ea00a3c40 0x7f6ea00a6100 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.630+0000 7f6ec056a640 1 --2- 172.21.15.46:0/46309026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6eb8103420 0x7f6eb81a01a0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.630+0000 7f6ec056a640 1 --2- 172.21.15.46:0/46309026 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6eb8102a60 0x7f6eb8199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.630+0000 7f6ec056a640 1 --2- 172.21.15.46:0/46309026 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6eb8101870 0x7f6eb8198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.630+0000 7f6ec056a640 1 -- 172.21.15.46:0/46309026 >> 172.21.15.46:0/46309026 conn(0x7f6eb80fcde0 msgr2=0x7f6eb8069ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.630+0000 7f6ec056a640 1 -- 172.21.15.46:0/46309026 shutdown_connections 2024-04-26T21:34:39.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.630+0000 7f6ec056a640 1 -- 172.21.15.46:0/46309026 wait complete. 2024-04-26T21:34:39.642 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T21:34:39.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.746+0000 7fab6688c640 1 -- 172.21.15.46:0/2699669728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab601033f0 msgr2=0x7fab601057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:39.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.746+0000 7fab6688c640 1 --2- 172.21.15.46:0/2699669728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab601033f0 0x7fab601057e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fab50009900 tx=0x7fab5002a410 comp rx=0 tx=0).stop 2024-04-26T21:34:39.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.746+0000 7fab6688c640 1 -- 172.21.15.46:0/2699669728 shutdown_connections 2024-04-26T21:34:39.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.746+0000 7fab6688c640 1 --2- 172.21.15.46:0/2699669728 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab60108650 0x7fab6010aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.746+0000 7fab6688c640 1 --2- 172.21.15.46:0/2699669728 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab60105d20 0x7fab60108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.746+0000 7fab6688c640 1 --2- 172.21.15.46:0/2699669728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab601033f0 0x7fab601057e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.746+0000 7fab6688c640 1 -- 172.21.15.46:0/2699669728 >> 172.21.15.46:0/2699669728 conn(0x7fab600fce20 msgr2=0x7fab600ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:39.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.746+0000 7fab6688c640 1 -- 172.21.15.46:0/2699669728 shutdown_connections 2024-04-26T21:34:39.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.746+0000 7fab6688c640 1 -- 172.21.15.46:0/2699669728 wait complete. 2024-04-26T21:34:39.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.747+0000 7fab6688c640 1 Processor -- start 2024-04-26T21:34:39.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.747+0000 7fab6688c640 1 -- start start 2024-04-26T21:34:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.747+0000 7fab6688c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab601033f0 0x7fab60198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.747+0000 7fab6688c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab60105d20 0x7fab601991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.747+0000 7fab6688c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab60108650 0x7fab601a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.747+0000 7fab6688c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fab6010e4d0 con 0x7fab60105d20 2024-04-26T21:34:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.747+0000 7fab6688c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fab6010e350 con 0x7fab60108650 2024-04-26T21:34:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.747+0000 7fab6688c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fab6010e650 con 0x7fab601033f0 2024-04-26T21:34:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.747+0000 7fab5f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab60105d20 0x7fab601991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.747+0000 7fab5ffff640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab601033f0 0x7fab60198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.747+0000 7fab5f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab60105d20 0x7fab601991a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41934/0 (socket says 172.21.15.46:41934) 2024-04-26T21:34:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.747+0000 7fab5f7fe640 1 -- 172.21.15.46:0/144188935 learned_addr learned my addr 172.21.15.46:0/144188935 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.748+0000 7fab64e29640 1 --2- 172.21.15.46:0/144188935 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab60108650 0x7fab601a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:39.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.748+0000 7fab5f7fe640 1 -- 172.21.15.46:0/144188935 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab601033f0 msgr2=0x7fab60198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:39.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.748+0000 7fab5f7fe640 1 --2- 172.21.15.46:0/144188935 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab601033f0 0x7fab60198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.748+0000 7fab5f7fe640 1 -- 172.21.15.46:0/144188935 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab60108650 msgr2=0x7fab601a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:39.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.748+0000 7fab5f7fe640 1 --2- 172.21.15.46:0/144188935 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab60108650 0x7fab601a01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.748+0000 7fab5f7fe640 1 -- 172.21.15.46:0/144188935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab601a0870 con 0x7fab60105d20 2024-04-26T21:34:39.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.748+0000 7fab5ffff640 1 --2- 172.21.15.46:0/144188935 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab601033f0 0x7fab60198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:34:39.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.748+0000 7fab5f7fe640 1 --2- 172.21.15.46:0/144188935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab60105d20 0x7fab601991a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fab5400e9f0 tx=0x7fab5400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:39.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.748+0000 7fab5d7fa640 1 -- 172.21.15.46:0/144188935 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab5400ccc0 con 0x7fab60105d20 2024-04-26T21:34:39.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.748+0000 7fab6688c640 1 -- 172.21.15.46:0/144188935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab601a0b60 con 0x7fab60105d20 2024-04-26T21:34:39.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.748+0000 7fab5d7fa640 1 -- 172.21.15.46:0/144188935 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab540054d0 con 0x7fab60105d20 2024-04-26T21:34:39.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.748+0000 7fab5d7fa640 1 -- 172.21.15.46:0/144188935 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab54010430 con 0x7fab60105d20 2024-04-26T21:34:39.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.748+0000 7fab6688c640 1 -- 172.21.15.46:0/144188935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab601a1060 con 0x7fab60105d20 2024-04-26T21:34:39.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.749+0000 7fab6688c640 1 -- 172.21.15.46:0/144188935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab600fe910 con 0x7fab60105d20 2024-04-26T21:34:39.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.750+0000 7fab5d7fa640 1 -- 172.21.15.46:0/144188935 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fab540046c0 con 0x7fab60105d20 2024-04-26T21:34:39.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.751+0000 7fab5d7fa640 1 --2- 172.21.15.46:0/144188935 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fab480a3c40 0x7fab480a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:39.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.751+0000 7fab5ffff640 1 --2- 172.21.15.46:0/144188935 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fab480a3c40 0x7fab480a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:39.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.751+0000 7fab5d7fa640 1 -- 172.21.15.46:0/144188935 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7fab540d09e0 con 0x7fab60105d20 2024-04-26T21:34:39.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.751+0000 7fab5ffff640 1 --2- 172.21.15.46:0/144188935 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fab480a3c40 0x7fab480a6100 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fab50009230 tx=0x7fab500314a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:39.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.755+0000 7fab5d7fa640 1 -- 172.21.15.46:0/144188935 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fab5409d290 con 0x7fab60105d20 2024-04-26T21:34:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.942+0000 7fab6688c640 1 -- 172.21.15.46:0/144188935 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fab6002dd20 con 0x7fab60105d20 2024-04-26T21:34:39.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.943+0000 7fab5d7fa640 1 -- 172.21.15.46:0/144188935 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7fab5409cb90 con 0x7fab60105d20 2024-04-26T21:34:39.944 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-26T21:34:39.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.945+0000 7fab6688c640 1 -- 172.21.15.46:0/144188935 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fab480a3c40 msgr2=0x7fab480a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:39.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.945+0000 7fab6688c640 1 --2- 172.21.15.46:0/144188935 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fab480a3c40 0x7fab480a6100 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fab50009230 tx=0x7fab500314a0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.945+0000 7fab6688c640 1 -- 172.21.15.46:0/144188935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab60105d20 msgr2=0x7fab601991a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:39.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.945+0000 7fab6688c640 1 --2- 172.21.15.46:0/144188935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab60105d20 0x7fab601991a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fab5400e9f0 tx=0x7fab5400eeb0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.945+0000 7fab6688c640 1 -- 172.21.15.46:0/144188935 shutdown_connections 2024-04-26T21:34:39.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.945+0000 7fab6688c640 1 --2- 172.21.15.46:0/144188935 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fab480a3c40 0x7fab480a6100 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.945+0000 7fab6688c640 1 --2- 172.21.15.46:0/144188935 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fab60108650 0x7fab601a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.945+0000 7fab6688c640 1 --2- 172.21.15.46:0/144188935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab60105d20 0x7fab601991a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.945+0000 7fab6688c640 1 --2- 172.21.15.46:0/144188935 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fab601033f0 0x7fab60198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:39.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.945+0000 7fab6688c640 1 -- 172.21.15.46:0/144188935 >> 172.21.15.46:0/144188935 conn(0x7fab600fce20 msgr2=0x7fab60103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:39.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.945+0000 7fab6688c640 1 -- 172.21.15.46:0/144188935 shutdown_connections 2024-04-26T21:34:39.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:39.945+0000 7fab6688c640 1 -- 172.21.15.46:0/144188935 wait complete. 2024-04-26T21:34:39.956 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:39 smithi046 ceph-mon[29201]: pgmap v351: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:39.956 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:39 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1955374774' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:34:39.956 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:39 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3983681706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:34:39.956 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:39 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1951226289' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:34:39.956 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:39 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/46309026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:34:39.957 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T21:34:40.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:39 smithi142 ceph-mon[32944]: pgmap v351: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:40.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:39 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/1955374774' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:34:40.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:39 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3983681706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:34:40.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:39 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/1951226289' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:34:40.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:39 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/46309026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:34:40.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.060+0000 7f660df51640 1 -- 172.21.15.46:0/2310802701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6608101890 msgr2=0x7f6608101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:40.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.060+0000 7f660df51640 1 --2- 172.21.15.46:0/2310802701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6608101890 0x7f6608101cb0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f65f8009900 tx=0x7f65f802a410 comp rx=0 tx=0).stop 2024-04-26T21:34:40.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.060+0000 7f660df51640 1 -- 172.21.15.46:0/2310802701 shutdown_connections 2024-04-26T21:34:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.060+0000 7f660df51640 1 --2- 172.21.15.46:0/2310802701 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6608103440 0x7f6608109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.060+0000 7f660df51640 1 --2- 172.21.15.46:0/2310802701 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6608102a80 0x7f6608102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.060+0000 7f660df51640 1 --2- 172.21.15.46:0/2310802701 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6608101890 0x7f6608101cb0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.060+0000 7f660df51640 1 -- 172.21.15.46:0/2310802701 >> 172.21.15.46:0/2310802701 conn(0x7f66080fce20 msgr2=0x7f66080ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.060+0000 7f660df51640 1 -- 172.21.15.46:0/2310802701 shutdown_connections 2024-04-26T21:34:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.060+0000 7f660df51640 1 -- 172.21.15.46:0/2310802701 wait complete. 2024-04-26T21:34:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.061+0000 7f660df51640 1 Processor -- start 2024-04-26T21:34:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.061+0000 7f660df51640 1 -- start start 2024-04-26T21:34:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.061+0000 7f660df51640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6608101890 0x7f6608198cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.061+0000 7f660df51640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6608102a80 0x7f6608199210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.061+0000 7f660df51640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6608103440 0x7f66081a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.061+0000 7f660df51640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f660810d400 con 0x7f6608102a80 2024-04-26T21:34:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.061+0000 7f660df51640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f660810d280 con 0x7f6608101890 2024-04-26T21:34:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.061+0000 7f660df51640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f660810d580 con 0x7f6608103440 2024-04-26T21:34:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.061+0000 7f6606ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6608102a80 0x7f6608199210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.062+0000 7f6606ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6608102a80 0x7f6608199210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41948/0 (socket says 172.21.15.46:41948) 2024-04-26T21:34:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.062+0000 7f66077fe640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6608101890 0x7f6608198cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.062+0000 7f6606ffd640 1 -- 172.21.15.46:0/1851570732 learned_addr learned my addr 172.21.15.46:0/1851570732 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.062+0000 7f6607fff640 1 --2- 172.21.15.46:0/1851570732 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6608103440 0x7f66081a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.062+0000 7f6606ffd640 1 -- 172.21.15.46:0/1851570732 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6608103440 msgr2=0x7f66081a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.062+0000 7f6606ffd640 1 --2- 172.21.15.46:0/1851570732 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6608103440 0x7f66081a01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.062+0000 7f6606ffd640 1 -- 172.21.15.46:0/1851570732 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6608101890 msgr2=0x7f6608198cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.062+0000 7f6606ffd640 1 --2- 172.21.15.46:0/1851570732 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6608101890 0x7f6608198cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.062+0000 7f6606ffd640 1 -- 172.21.15.46:0/1851570732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66081a0840 con 0x7f6608102a80 2024-04-26T21:34:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.062+0000 7f6606ffd640 1 --2- 172.21.15.46:0/1851570732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6608102a80 0x7f6608199210 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f65fc00d770 tx=0x7f65fc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.062+0000 7f6604ff9640 1 -- 172.21.15.46:0/1851570732 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65fc017070 con 0x7f6608102a80 2024-04-26T21:34:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.062+0000 7f6604ff9640 1 -- 172.21.15.46:0/1851570732 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65fc0054d0 con 0x7f6608102a80 2024-04-26T21:34:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.062+0000 7f660df51640 1 -- 172.21.15.46:0/1851570732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66081a0b30 con 0x7f6608102a80 2024-04-26T21:34:40.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.062+0000 7f6604ff9640 1 -- 172.21.15.46:0/1851570732 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65fc004c30 con 0x7f6608102a80 2024-04-26T21:34:40.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.062+0000 7f660df51640 1 -- 172.21.15.46:0/1851570732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66081a1000 con 0x7f6608102a80 2024-04-26T21:34:40.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.063+0000 7f660df51640 1 -- 172.21.15.46:0/1851570732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66081a11d0 con 0x7f6608102a80 2024-04-26T21:34:40.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.064+0000 7f6604ff9640 1 -- 172.21.15.46:0/1851570732 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f65fc004e10 con 0x7f6608102a80 2024-04-26T21:34:40.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.065+0000 7f6604ff9640 1 --2- 172.21.15.46:0/1851570732 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f65f00a3c40 0x7f65f00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:40.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.065+0000 7f66077fe640 1 --2- 172.21.15.46:0/1851570732 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f65f00a3c40 0x7f65f00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:40.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.065+0000 7f6604ff9640 1 -- 172.21.15.46:0/1851570732 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f65fc0d0000 con 0x7f6608102a80 2024-04-26T21:34:40.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.065+0000 7f66077fe640 1 --2- 172.21.15.46:0/1851570732 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f65f00a3c40 0x7f65f00a6100 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f65f8005ab0 tx=0x7f65f8005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:40.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.069+0000 7f6604ff9640 1 -- 172.21.15.46:0/1851570732 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f65fc09c260 con 0x7f6608102a80 2024-04-26T21:34:40.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:39 smithi050 ceph-mon[32753]: pgmap v351: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:40.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:39 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/1955374774' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:34:40.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:39 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3983681706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:34:40.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:39 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/1951226289' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:34:40.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:39 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/46309026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:34:40.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.255+0000 7f660df51640 1 -- 172.21.15.46:0/1851570732 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f66081a1560 con 0x7f6608102a80 2024-04-26T21:34:40.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.256+0000 7f6604ff9640 1 -- 172.21.15.46:0/1851570732 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f65fc0a0560 con 0x7f6608102a80 2024-04-26T21:34:40.257 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-26T21:34:40.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.258+0000 7f660df51640 1 -- 172.21.15.46:0/1851570732 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f65f00a3c40 msgr2=0x7f65f00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:40.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.258+0000 7f660df51640 1 --2- 172.21.15.46:0/1851570732 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f65f00a3c40 0x7f65f00a6100 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f65f8005ab0 tx=0x7f65f8005a20 comp rx=0 tx=0).stop 2024-04-26T21:34:40.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.258+0000 7f660df51640 1 -- 172.21.15.46:0/1851570732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6608102a80 msgr2=0x7f6608199210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:40.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.258+0000 7f660df51640 1 --2- 172.21.15.46:0/1851570732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6608102a80 0x7f6608199210 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f65fc00d770 tx=0x7f65fc00dc30 comp rx=0 tx=0).stop 2024-04-26T21:34:40.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.258+0000 7f660df51640 1 -- 172.21.15.46:0/1851570732 shutdown_connections 2024-04-26T21:34:40.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.258+0000 7f660df51640 1 --2- 172.21.15.46:0/1851570732 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f65f00a3c40 0x7f65f00a6100 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.258+0000 7f660df51640 1 --2- 172.21.15.46:0/1851570732 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6608103440 0x7f66081a01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.258+0000 7f660df51640 1 --2- 172.21.15.46:0/1851570732 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6608102a80 0x7f6608199210 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.258+0000 7f660df51640 1 --2- 172.21.15.46:0/1851570732 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6608101890 0x7f6608198cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.258+0000 7f660df51640 1 -- 172.21.15.46:0/1851570732 >> 172.21.15.46:0/1851570732 conn(0x7f66080fce20 msgr2=0x7f66080fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:40.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.258+0000 7f660df51640 1 -- 172.21.15.46:0/1851570732 shutdown_connections 2024-04-26T21:34:40.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.258+0000 7f660df51640 1 -- 172.21.15.46:0/1851570732 wait complete. 2024-04-26T21:34:40.270 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:34:40.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.371+0000 7f97d0c93640 1 -- 172.21.15.46:0/1946053929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97cc101870 msgr2=0x7f97cc101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:40.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.371+0000 7f97d0c93640 1 --2- 172.21.15.46:0/1946053929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97cc101870 0x7f97cc101c90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f97bc009ca0 tx=0x7f97bc02a8c0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.371+0000 7f97d0c93640 1 -- 172.21.15.46:0/1946053929 shutdown_connections 2024-04-26T21:34:40.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.371+0000 7f97d0c93640 1 --2- 172.21.15.46:0/1946053929 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97cc103420 0x7f97cc109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.371+0000 7f97d0c93640 1 --2- 172.21.15.46:0/1946053929 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97cc102a60 0x7f97cc102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.371+0000 7f97d0c93640 1 --2- 172.21.15.46:0/1946053929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97cc101870 0x7f97cc101c90 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.371+0000 7f97d0c93640 1 -- 172.21.15.46:0/1946053929 >> 172.21.15.46:0/1946053929 conn(0x7f97cc0fcde0 msgr2=0x7f97cc0ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:40.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.371+0000 7f97d0c93640 1 -- 172.21.15.46:0/1946053929 shutdown_connections 2024-04-26T21:34:40.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.371+0000 7f97d0c93640 1 -- 172.21.15.46:0/1946053929 wait complete. 2024-04-26T21:34:40.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.372+0000 7f97d0c93640 1 Processor -- start 2024-04-26T21:34:40.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.372+0000 7f97d0c93640 1 -- start start 2024-04-26T21:34:40.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.372+0000 7f97d0c93640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97cc101870 0x7f97cc105800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:40.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.372+0000 7f97d0c93640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97cc102a60 0x7f97cc103e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:40.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.372+0000 7f97d0c93640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97cc103420 0x7f97cc104380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:40.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.372+0000 7f97d0c93640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f97cc10d3a0 con 0x7f97cc101870 2024-04-26T21:34:40.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.372+0000 7f97d0c93640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f97cc10d220 con 0x7f97cc103420 2024-04-26T21:34:40.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.372+0000 7f97d0c93640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f97cc10d520 con 0x7f97cc102a60 2024-04-26T21:34:40.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.372+0000 7f97ca59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97cc101870 0x7f97cc105800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:40.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.372+0000 7f97ca59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97cc101870 0x7f97cc105800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41980/0 (socket says 172.21.15.46:41980) 2024-04-26T21:34:40.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.372+0000 7f97ca59c640 1 -- 172.21.15.46:0/3119178788 learned_addr learned my addr 172.21.15.46:0/3119178788 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:40.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.372+0000 7f97c9d9b640 1 --2- 172.21.15.46:0/3119178788 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97cc102a60 0x7f97cc103e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:40.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.372+0000 7f97cad9d640 1 --2- 172.21.15.46:0/3119178788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97cc103420 0x7f97cc104380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:40.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.372+0000 7f97ca59c640 1 -- 172.21.15.46:0/3119178788 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97cc102a60 msgr2=0x7f97cc103e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:40.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.372+0000 7f97ca59c640 1 --2- 172.21.15.46:0/3119178788 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97cc102a60 0x7f97cc103e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.372+0000 7f97ca59c640 1 -- 172.21.15.46:0/3119178788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97cc103420 msgr2=0x7f97cc104380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:40.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.372+0000 7f97ca59c640 1 --2- 172.21.15.46:0/3119178788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97cc103420 0x7f97cc104380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.372+0000 7f97ca59c640 1 -- 172.21.15.46:0/3119178788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97cc104c10 con 0x7f97cc101870 2024-04-26T21:34:40.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.372+0000 7f97c9d9b640 1 --2- 172.21.15.46:0/3119178788 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97cc102a60 0x7f97cc103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:34:40.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.372+0000 7f97cad9d640 1 --2- 172.21.15.46:0/3119178788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97cc103420 0x7f97cc104380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:34:40.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.372+0000 7f97ca59c640 1 --2- 172.21.15.46:0/3119178788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97cc101870 0x7f97cc105800 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f97bc009430 tx=0x7f97bc031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:40.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.373+0000 7f97bb7fe640 1 -- 172.21.15.46:0/3119178788 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97bc003cb0 con 0x7f97cc101870 2024-04-26T21:34:40.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.373+0000 7f97bb7fe640 1 -- 172.21.15.46:0/3119178788 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97bc003e50 con 0x7f97cc101870 2024-04-26T21:34:40.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.373+0000 7f97d0c93640 1 -- 172.21.15.46:0/3119178788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97cc1a2d10 con 0x7f97cc101870 2024-04-26T21:34:40.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.373+0000 7f97bb7fe640 1 -- 172.21.15.46:0/3119178788 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97bc02fc60 con 0x7f97cc101870 2024-04-26T21:34:40.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.373+0000 7f97d0c93640 1 -- 172.21.15.46:0/3119178788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97cc1a31b0 con 0x7f97cc101870 2024-04-26T21:34:40.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.374+0000 7f97d0c93640 1 -- 172.21.15.46:0/3119178788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97cc1a3380 con 0x7f97cc101870 2024-04-26T21:34:40.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.375+0000 7f97bb7fe640 1 -- 172.21.15.46:0/3119178788 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f97bc004040 con 0x7f97cc101870 2024-04-26T21:34:40.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.375+0000 7f97bb7fe640 1 --2- 172.21.15.46:0/3119178788 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f97b40a3c40 0x7f97b40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:40.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.375+0000 7f97c9d9b640 1 --2- 172.21.15.46:0/3119178788 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f97b40a3c40 0x7f97b40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:40.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.375+0000 7f97bb7fe640 1 -- 172.21.15.46:0/3119178788 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f97bc0f1ca0 con 0x7f97cc101870 2024-04-26T21:34:40.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.376+0000 7f97c9d9b640 1 --2- 172.21.15.46:0/3119178788 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f97b40a3c40 0x7f97b40a6100 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f97c0005d00 tx=0x7f97c0005c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:40.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.379+0000 7f97bb7fe640 1 -- 172.21.15.46:0/3119178788 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f97bc0be660 con 0x7f97cc101870 2024-04-26T21:34:40.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.605+0000 7f97d0c93640 1 -- 172.21.15.46:0/3119178788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97cc1a3550 con 0x7f97cc101870 2024-04-26T21:34:40.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.606+0000 7f97bb7fe640 1 -- 172.21.15.46:0/3119178788 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+11863 (secure 0 0 0) 0x7f97bc0bdf60 con 0x7f97cc101870 2024-04-26T21:34:40.607 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:34:40.607 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3674296286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3674296286},{"type":"v1","addr":"172.21.15.46:6835","nonce":3674296286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/391791206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":391791206},{"type":"v1","addr":"172.21.15.46:6837","nonce":391791206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/1222489912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1222489912},{"type":"v1","addr":"172.21.15.46:6839","nonce":1222489912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14658,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/860605548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":860605548},{"type":"v1","addr":"172.21.15.46:6841","nonce":860605548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3621399933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3621399933},{"type":"v1","addr":"172.21.15.142:6833","nonce":3621399933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/345015778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":345015778},{"type":"v1","addr":"172.21.15.50:6833","nonce":345015778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T21:32:44.201541+0000","modified":"2024-04-26T21:32:49.889029+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24526,"mds_2":24509},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/2546509863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":2546509863},{"type":"v1","addr":"172.21.15.50:6835","nonce":2546509863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/3791468066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":3791468066},{"type":"v1","addr":"172.21.15.142:6835","nonce":3791468066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6837/1723132553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1723132553},{"type":"v1","addr":"172.21.15.142:6837","nonce":1723132553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6837/3189620395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3189620395},{"type":"v1","addr":"172.21.15.50:6837","nonce":3189620395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24526":{"gid":24526,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.142:6839/3089509101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3089509101},{"type":"v1","addr":"172.21.15.142:6839","nonce":3089509101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6839/2453253368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2453253368},{"type":"v1","addr":"172.21.15.50:6839","nonce":2453253368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24509,24526,24539]},"id":1}]} 2024-04-26T21:34:40.608 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 22 2024-04-26T21:34:40.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.608+0000 7f97d0c93640 1 -- 172.21.15.46:0/3119178788 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f97b40a3c40 msgr2=0x7f97b40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:40.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.608+0000 7f97d0c93640 1 --2- 172.21.15.46:0/3119178788 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f97b40a3c40 0x7f97b40a6100 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f97c0005d00 tx=0x7f97c0005c70 comp rx=0 tx=0).stop 2024-04-26T21:34:40.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.608+0000 7f97d0c93640 1 -- 172.21.15.46:0/3119178788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97cc101870 msgr2=0x7f97cc105800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:40.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.608+0000 7f97d0c93640 1 --2- 172.21.15.46:0/3119178788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97cc101870 0x7f97cc105800 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f97bc009430 tx=0x7f97bc031840 comp rx=0 tx=0).stop 2024-04-26T21:34:40.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.608+0000 7f97d0c93640 1 -- 172.21.15.46:0/3119178788 shutdown_connections 2024-04-26T21:34:40.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.608+0000 7f97d0c93640 1 --2- 172.21.15.46:0/3119178788 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f97b40a3c40 0x7f97b40a6100 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.608+0000 7f97d0c93640 1 --2- 172.21.15.46:0/3119178788 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97cc103420 0x7f97cc104380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.608+0000 7f97d0c93640 1 --2- 172.21.15.46:0/3119178788 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f97cc102a60 0x7f97cc103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.608+0000 7f97d0c93640 1 --2- 172.21.15.46:0/3119178788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97cc101870 0x7f97cc105800 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.608+0000 7f97d0c93640 1 -- 172.21.15.46:0/3119178788 >> 172.21.15.46:0/3119178788 conn(0x7f97cc0fcde0 msgr2=0x7f97cc069bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:40.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.608+0000 7f97d0c93640 1 -- 172.21.15.46:0/3119178788 shutdown_connections 2024-04-26T21:34:40.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.608+0000 7f97d0c93640 1 -- 172.21.15.46:0/3119178788 wait complete. 2024-04-26T21:34:40.621 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:34:40.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.722+0000 7f24d37c1640 1 -- 172.21.15.46:0/3357359576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24cc104550 msgr2=0x7f24cc10ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:40.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.722+0000 7f24d37c1640 1 --2- 172.21.15.46:0/3357359576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24cc104550 0x7f24cc10ade0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f24c4009ca0 tx=0x7f24c402a8c0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.722+0000 7f24d37c1640 1 -- 172.21.15.46:0/3357359576 shutdown_connections 2024-04-26T21:34:40.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.722+0000 7f24d37c1640 1 --2- 172.21.15.46:0/3357359576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24cc104550 0x7f24cc10ade0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.722+0000 7f24d37c1640 1 --2- 172.21.15.46:0/3357359576 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24cc103b90 0x7f24cc104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.722+0000 7f24d37c1640 1 --2- 172.21.15.46:0/3357359576 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24cc1029a0 0x7f24cc102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.722+0000 7f24d37c1640 1 -- 172.21.15.46:0/3357359576 >> 172.21.15.46:0/3357359576 conn(0x7f24cc0fdf30 msgr2=0x7f24cc100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:40.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.723+0000 7f24d37c1640 1 -- 172.21.15.46:0/3357359576 shutdown_connections 2024-04-26T21:34:40.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.723+0000 7f24d37c1640 1 -- 172.21.15.46:0/3357359576 wait complete. 2024-04-26T21:34:40.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.723+0000 7f24d37c1640 1 Processor -- start 2024-04-26T21:34:40.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.723+0000 7f24d37c1640 1 -- start start 2024-04-26T21:34:40.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.723+0000 7f24d37c1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24cc1029a0 0x7f24cc198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:40.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.723+0000 7f24d37c1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24cc103b90 0x7f24cc1990b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:40.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.723+0000 7f24d37c1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24cc104550 0x7f24cc1a0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:40.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.723+0000 7f24d37c1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc10e4d0 con 0x7f24cc104550 2024-04-26T21:34:40.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.723+0000 7f24d37c1640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc10e350 con 0x7f24cc103b90 2024-04-26T21:34:40.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.723+0000 7f24d37c1640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc10e650 con 0x7f24cc1029a0 2024-04-26T21:34:40.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.723+0000 7f24d1d5e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24cc104550 0x7f24cc1a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:40.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.723+0000 7f24d1d5e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24cc104550 0x7f24cc1a0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41992/0 (socket says 172.21.15.46:41992) 2024-04-26T21:34:40.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.723+0000 7f24d1d5e640 1 -- 172.21.15.46:0/1367623893 learned_addr learned my addr 172.21.15.46:0/1367623893 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:40.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.723+0000 7f24d155d640 1 --2- 172.21.15.46:0/1367623893 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24cc1029a0 0x7f24cc198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:40.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.724+0000 7f24d0d5c640 1 --2- 172.21.15.46:0/1367623893 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24cc103b90 0x7f24cc1990b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:40.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.724+0000 7f24d1d5e640 1 -- 172.21.15.46:0/1367623893 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24cc1029a0 msgr2=0x7f24cc198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:40.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.724+0000 7f24d1d5e640 1 --2- 172.21.15.46:0/1367623893 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24cc1029a0 0x7f24cc198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.724+0000 7f24d1d5e640 1 -- 172.21.15.46:0/1367623893 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24cc103b90 msgr2=0x7f24cc1990b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:40.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.724+0000 7f24d1d5e640 1 --2- 172.21.15.46:0/1367623893 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24cc103b90 0x7f24cc1990b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.724+0000 7f24d1d5e640 1 -- 172.21.15.46:0/1367623893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24cc1a0790 con 0x7f24cc104550 2024-04-26T21:34:40.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.724+0000 7f24d1d5e640 1 --2- 172.21.15.46:0/1367623893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24cc104550 0x7f24cc1a0120 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f24c40097b0 tx=0x7f24c4003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:40.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.724+0000 7f24c27fc640 1 -- 172.21.15.46:0/1367623893 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c4004240 con 0x7f24cc104550 2024-04-26T21:34:40.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.724+0000 7f24c27fc640 1 -- 172.21.15.46:0/1367623893 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24c40317c0 con 0x7f24cc104550 2024-04-26T21:34:40.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.724+0000 7f24d37c1640 1 -- 172.21.15.46:0/1367623893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24cc1a0a20 con 0x7f24cc104550 2024-04-26T21:34:40.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.724+0000 7f24c27fc640 1 -- 172.21.15.46:0/1367623893 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c402fd80 con 0x7f24cc104550 2024-04-26T21:34:40.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.724+0000 7f24d37c1640 1 -- 172.21.15.46:0/1367623893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24cc1a0f70 con 0x7f24cc104550 2024-04-26T21:34:40.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.725+0000 7f24d37c1640 1 -- 172.21.15.46:0/1367623893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24cc109d90 con 0x7f24cc104550 2024-04-26T21:34:40.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.726+0000 7f24c27fc640 1 -- 172.21.15.46:0/1367623893 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f24c40043e0 con 0x7f24cc104550 2024-04-26T21:34:40.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.727+0000 7f24c27fc640 1 --2- 172.21.15.46:0/1367623893 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f24b80a3c40 0x7f24b80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:40.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.727+0000 7f24d155d640 1 --2- 172.21.15.46:0/1367623893 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f24b80a3c40 0x7f24b80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:40.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.727+0000 7f24c27fc640 1 -- 172.21.15.46:0/1367623893 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f24c40f1fc0 con 0x7f24cc104550 2024-04-26T21:34:40.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.727+0000 7f24d155d640 1 --2- 172.21.15.46:0/1367623893 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f24b80a3c40 0x7f24b80a6100 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f24c80058c0 tx=0x7f24c800a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:40.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.731+0000 7f24c27fc640 1 -- 172.21.15.46:0/1367623893 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f24c40beb40 con 0x7f24cc104550 2024-04-26T21:34:40.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.917+0000 7f24d37c1640 1 -- 172.21.15.46:0/1367623893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f24cc02dc50 con 0x7f24cc104550 2024-04-26T21:34:40.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.918+0000 7f24c27fc640 1 -- 172.21.15.46:0/1367623893 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22602 (secure 0 0 0) 0x7f24c40be440 con 0x7f24cc104550 2024-04-26T21:34:40.919 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:34:40.920 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":87,"fsid":"36c7013a-0413-11ef-bc93-c7b262605968","created":"2024-04-26T21:24:30.481156+0000","modified":"2024-04-26T21:33:31.051554+0000","last_up_change":"2024-04-26T21:31:50.100491+0000","last_in_change":"2024-04-26T21:31:29.643428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:27:57.935988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:32:42.195211+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:32:43.162114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f53c8702-e4da-435f-a9dc-3911d0d36ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6803","nonce":2762665125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6805","nonce":2762665125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6809","nonce":2762665125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6807","nonce":2762665125}]},"public_addr":"172.21.15.46:6803/2762665125","cluster_addr":"172.21.15.46:6805/2762665125","heartbeat_back_addr":"172.21.15.46:6809/2762665125","heartbeat_front_addr":"172.21.15.46:6807/2762665125","state":["exists","up"]},{"osd":1,"uuid":"fc4d6f52-fc8a-4efd-80c9-c092b57e7f42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6801","nonce":1628008970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6803","nonce":1628008970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6807","nonce":1628008970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6805","nonce":1628008970}]},"public_addr":"172.21.15.50:6801/1628008970","cluster_addr":"172.21.15.50:6803/1628008970","heartbeat_back_addr":"172.21.15.50:6807/1628008970","heartbeat_front_addr":"172.21.15.50:6805/1628008970","state":["exists","up"]},{"osd":2,"uuid":"26d1baa1-4ea1-4b7e-9fcf-4f8bbf7329c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6801","nonce":1513279137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6803","nonce":1513279137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6807","nonce":1513279137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6805","nonce":1513279137}]},"public_addr":"172.21.15.142:6801/1513279137","cluster_addr":"172.21.15.142:6803/1513279137","heartbeat_back_addr":"172.21.15.142:6807/1513279137","heartbeat_front_addr":"172.21.15.142:6805/1513279137","state":["exists","up"]},{"osd":3,"uuid":"34d2401f-5fcc-491c-8fcf-c3c492611f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6811","nonce":2435181760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6813","nonce":2435181760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6817","nonce":2435181760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6815","nonce":2435181760}]},"public_addr":"172.21.15.46:6811/2435181760","cluster_addr":"172.21.15.46:6813/2435181760","heartbeat_back_addr":"172.21.15.46:6817/2435181760","heartbeat_front_addr":"172.21.15.46:6815/2435181760","state":["exists","up"]},{"osd":4,"uuid":"986bb709-e4cf-400d-982e-c3f5ac413e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6809","nonce":3177333250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6811","nonce":3177333250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6815","nonce":3177333250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6813","nonce":3177333250}]},"public_addr":"172.21.15.50:6809/3177333250","cluster_addr":"172.21.15.50:6811/3177333250","heartbeat_back_addr":"172.21.15.50:6815/3177333250","heartbeat_front_addr":"172.21.15.50:6813/3177333250","state":["exists","up"]},{"osd":5,"uuid":"b7d65851-ebe1-45e2-833d-8022446f2698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6809","nonce":4113107737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6811","nonce":4113107737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6815","nonce":4113107737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6813","nonce":4113107737}]},"public_addr":"172.21.15.142:6809/4113107737","cluster_addr":"172.21.15.142:6811/4113107737","heartbeat_back_addr":"172.21.15.142:6815/4113107737","heartbeat_front_addr":"172.21.15.142:6813/4113107737","state":["exists","up"]},{"osd":6,"uuid":"2a3e96d7-d5bf-44ff-9134-29f1420c0e4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6819","nonce":2325162882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6821","nonce":2325162882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6825","nonce":2325162882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6823","nonce":2325162882}]},"public_addr":"172.21.15.46:6819/2325162882","cluster_addr":"172.21.15.46:6821/2325162882","heartbeat_back_addr":"172.21.15.46:6825/2325162882","heartbeat_front_addr":"172.21.15.46:6823/2325162882","state":["exists","up"]},{"osd":7,"uuid":"e53ae885-23b6-46fd-b4b1-c62f31a6921f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6817","nonce":820310228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6819","nonce":820310228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6823","nonce":820310228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6821","nonce":820310228}]},"public_addr":"172.21.15.50:6817/820310228","cluster_addr":"172.21.15.50:6819/820310228","heartbeat_back_addr":"172.21.15.50:6823/820310228","heartbeat_front_addr":"172.21.15.50:6821/820310228","state":["exists","up"]},{"osd":8,"uuid":"356e2692-16a9-4045-b58f-c8f9cb5d7241","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6817","nonce":1824918711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6819","nonce":1824918711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6823","nonce":1824918711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6821","nonce":1824918711}]},"public_addr":"172.21.15.142:6817/1824918711","cluster_addr":"172.21.15.142:6819/1824918711","heartbeat_back_addr":"172.21.15.142:6823/1824918711","heartbeat_front_addr":"172.21.15.142:6821/1824918711","state":["exists","up"]},{"osd":9,"uuid":"8e55570e-77e5-4892-8b9f-77dcd03a4eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6827","nonce":3601947736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6829","nonce":3601947736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6833","nonce":3601947736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6831","nonce":3601947736}]},"public_addr":"172.21.15.46:6827/3601947736","cluster_addr":"172.21.15.46:6829/3601947736","heartbeat_back_addr":"172.21.15.46:6833/3601947736","heartbeat_front_addr":"172.21.15.46:6831/3601947736","state":["exists","up"]},{"osd":10,"uuid":"366389fa-fb9a-408b-b5c2-343c054c5a64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6825","nonce":3135765006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6827","nonce":3135765006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6831","nonce":3135765006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6829","nonce":3135765006}]},"public_addr":"172.21.15.50:6825/3135765006","cluster_addr":"172.21.15.50:6827/3135765006","heartbeat_back_addr":"172.21.15.50:6831/3135765006","heartbeat_front_addr":"172.21.15.50:6829/3135765006","state":["exists","up"]},{"osd":11,"uuid":"7b6e3d93-59e1-4853-b4a2-261b728c47a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6825","nonce":1482712612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6827","nonce":1482712612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6831","nonce":1482712612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6829","nonce":1482712612}]},"public_addr":"172.21.15.142:6825/1482712612","cluster_addr":"172.21.15.142:6827/1482712612","heartbeat_back_addr":"172.21.15.142:6831/1482712612","heartbeat_front_addr":"172.21.15.142:6829/1482712612","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:08.082015+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:32.634928+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:55.978864+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:21.796863+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:46.862518+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:11.440432+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:37.764777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:03.374958+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:28.714052+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:55.761167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:22.330951+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:47.839334+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/2465242088":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6801/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6800/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/1381997090":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/3507268217":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2570024356":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/2691061938":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2775954801":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6800/2896098005":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6801/2896098005":"2024-04-27T21:25:02.078730+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:34:40.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.920+0000 7f24d37c1640 1 -- 172.21.15.46:0/1367623893 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f24b80a3c40 msgr2=0x7f24b80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:40.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.920+0000 7f24d37c1640 1 --2- 172.21.15.46:0/1367623893 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f24b80a3c40 0x7f24b80a6100 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f24c80058c0 tx=0x7f24c800a2d0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.920+0000 7f24d37c1640 1 -- 172.21.15.46:0/1367623893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24cc104550 msgr2=0x7f24cc1a0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:40.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.920+0000 7f24d37c1640 1 --2- 172.21.15.46:0/1367623893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24cc104550 0x7f24cc1a0120 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f24c40097b0 tx=0x7f24c4003c20 comp rx=0 tx=0).stop 2024-04-26T21:34:40.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.920+0000 7f24d37c1640 1 -- 172.21.15.46:0/1367623893 shutdown_connections 2024-04-26T21:34:40.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.920+0000 7f24d37c1640 1 --2- 172.21.15.46:0/1367623893 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f24b80a3c40 0x7f24b80a6100 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.920+0000 7f24d37c1640 1 --2- 172.21.15.46:0/1367623893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f24cc104550 0x7f24cc1a0120 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.920+0000 7f24d37c1640 1 --2- 172.21.15.46:0/1367623893 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24cc103b90 0x7f24cc1990b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.920+0000 7f24d37c1640 1 --2- 172.21.15.46:0/1367623893 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f24cc1029a0 0x7f24cc198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:40.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.920+0000 7f24d37c1640 1 -- 172.21.15.46:0/1367623893 >> 172.21.15.46:0/1367623893 conn(0x7f24cc0fdf30 msgr2=0x7f24cc106f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:40.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.920+0000 7f24d37c1640 1 -- 172.21.15.46:0/1367623893 shutdown_connections 2024-04-26T21:34:40.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:40.920+0000 7f24d37c1640 1 -- 172.21.15.46:0/1367623893 wait complete. 2024-04-26T21:34:40.931 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:40 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/144188935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:34:40.931 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:40 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1851570732' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:34:40.931 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:40 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3119178788' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:34:40.933 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T21:32:44.201541+0000', 'modified': '2024-04-26T21:32:49.889029+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24539, 'mds_1': 24526, 'mds_2': 24509}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24470': {'gid': 24470, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.50:6835/2546509863', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6834', 'nonce': 2546509863}, {'type': 'v1', 'addr': '172.21.15.50:6835', 'nonce': 2546509863}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24493': {'gid': 24493, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.142:6835/3791468066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6834', 'nonce': 3791468066}, {'type': 'v1', 'addr': '172.21.15.142:6835', 'nonce': 3791468066}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24505': {'gid': 24505, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.142:6837/1723132553', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6836', 'nonce': 1723132553}, {'type': 'v1', 'addr': '172.21.15.142:6837', 'nonce': 1723132553}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24509': {'gid': 24509, 'name': 'h', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.50:6837/3189620395', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6836', 'nonce': 3189620395}, {'type': 'v1', 'addr': '172.21.15.50:6837', 'nonce': 3189620395}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24526': {'gid': 24526, 'name': 'i', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.142:6839/3089509101', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.142:6838', 'nonce': 3089509101}, {'type': 'v1', 'addr': '172.21.15.142:6839', 'nonce': 3089509101}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24539': {'gid': 24539, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.50:6839/2453253368', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6838', 'nonce': 2453253368}, {'type': 'v1', 'addr': '172.21.15.50:6839', 'nonce': 2453253368}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24539, 'qdb_cluster': [24509, 24526, 24539]} 2024-04-26T21:34:40.933 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T21:34:40.933 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-26T21:34:41.019 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:40 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/144188935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:34:41.019 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:40 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/1851570732' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:34:41.019 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:40 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3119178788' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:34:41.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.036+0000 7feb30808640 1 -- 172.21.15.46:0/1246845304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb2c102500 msgr2=0x7feb2c102980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.036+0000 7feb30808640 1 --2- 172.21.15.46:0/1246845304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb2c102500 0x7feb2c102980 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7feb24009ca0 tx=0x7feb2402a8c0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.036+0000 7feb30808640 1 -- 172.21.15.46:0/1246845304 shutdown_connections 2024-04-26T21:34:41.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.036+0000 7feb30808640 1 --2- 172.21.15.46:0/1246845304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb2c102500 0x7feb2c102980 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.036+0000 7feb30808640 1 --2- 172.21.15.46:0/1246845304 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb2c101b60 0x7feb2c101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.036+0000 7feb30808640 1 --2- 172.21.15.46:0/1246845304 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feb2c101200 0x7feb2c101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.036+0000 7feb30808640 1 -- 172.21.15.46:0/1246845304 >> 172.21.15.46:0/1246845304 conn(0x7feb2c0fce20 msgr2=0x7feb2c0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:41.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.036+0000 7feb30808640 1 -- 172.21.15.46:0/1246845304 shutdown_connections 2024-04-26T21:34:41.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.036+0000 7feb30808640 1 -- 172.21.15.46:0/1246845304 wait complete. 2024-04-26T21:34:41.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.037+0000 7feb30808640 1 Processor -- start 2024-04-26T21:34:41.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.037+0000 7feb30808640 1 -- start start 2024-04-26T21:34:41.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.037+0000 7feb30808640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feb2c101200 0x7feb2c19ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.037+0000 7feb30808640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb2c101b60 0x7feb2c19b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.037+0000 7feb30808640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb2c102500 0x7feb2c1a23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.037+0000 7feb30808640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7feb2c110700 con 0x7feb2c101b60 2024-04-26T21:34:41.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.037+0000 7feb30808640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7feb2c110580 con 0x7feb2c102500 2024-04-26T21:34:41.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.037+0000 7feb30808640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7feb2c110880 con 0x7feb2c101200 2024-04-26T21:34:41.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.037+0000 7feb23fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb2c101b60 0x7feb2c19b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.037+0000 7feb23fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb2c101b60 0x7feb2c19b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42020/0 (socket says 172.21.15.46:42020) 2024-04-26T21:34:41.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.037+0000 7feb23fff640 1 -- 172.21.15.46:0/836376564 learned_addr learned my addr 172.21.15.46:0/836376564 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:41.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.037+0000 7feb2ad9d640 1 --2- 172.21.15.46:0/836376564 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb2c102500 0x7feb2c1a23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.037+0000 7feb2a59c640 1 --2- 172.21.15.46:0/836376564 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feb2c101200 0x7feb2c19ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.037+0000 7feb23fff640 1 -- 172.21.15.46:0/836376564 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feb2c101200 msgr2=0x7feb2c19ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.037+0000 7feb23fff640 1 --2- 172.21.15.46:0/836376564 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feb2c101200 0x7feb2c19ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.037+0000 7feb23fff640 1 -- 172.21.15.46:0/836376564 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb2c102500 msgr2=0x7feb2c1a23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.037+0000 7feb23fff640 1 --2- 172.21.15.46:0/836376564 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb2c102500 0x7feb2c1a23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.037+0000 7feb23fff640 1 -- 172.21.15.46:0/836376564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb2c1a2a20 con 0x7feb2c101b60 2024-04-26T21:34:41.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.038+0000 7feb23fff640 1 --2- 172.21.15.46:0/836376564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb2c101b60 0x7feb2c19b340 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7feb1400c890 tx=0x7feb1400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:41.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.038+0000 7feb237fe640 1 -- 172.21.15.46:0/836376564 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb14004220 con 0x7feb2c101b60 2024-04-26T21:34:41.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.038+0000 7feb237fe640 1 -- 172.21.15.46:0/836376564 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb140043c0 con 0x7feb2c101b60 2024-04-26T21:34:41.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.038+0000 7feb30808640 1 -- 172.21.15.46:0/836376564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb2c1a2d10 con 0x7feb2c101b60 2024-04-26T21:34:41.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.038+0000 7feb237fe640 1 -- 172.21.15.46:0/836376564 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb14004a00 con 0x7feb2c101b60 2024-04-26T21:34:41.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.038+0000 7feb30808640 1 -- 172.21.15.46:0/836376564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb2c1a3280 con 0x7feb2c101b60 2024-04-26T21:34:41.043 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:40 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/144188935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:34:41.043 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:40 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/1851570732' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:34:41.043 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:40 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3119178788' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:34:41.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.040+0000 7feb237fe640 1 -- 172.21.15.46:0/836376564 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7feb140079b0 con 0x7feb2c101b60 2024-04-26T21:34:41.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.040+0000 7feb335f5740 1 Processor -- start 2024-04-26T21:34:41.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.040+0000 7feb335f5740 1 -- start start 2024-04-26T21:34:41.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.040+0000 7feb335f5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564cdf223690 0x564cdf223ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.041+0000 7feb335f5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564cdf224080 0x564cdf228b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.041+0000 7feb335f5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cdf2291f0 0x564cdf22b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.041+0000 7feb335f5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564cdf188740 con 0x564cdf2291f0 2024-04-26T21:34:41.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.041+0000 7feb335f5740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x564cdf1243a0 con 0x564cdf224080 2024-04-26T21:34:41.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.041+0000 7feb335f5740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x564cdf184700 con 0x564cdf223690 2024-04-26T21:34:41.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.041+0000 7feb237fe640 1 --2- 172.21.15.46:0/836376564 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7feb180a3c40 0x7feb180a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.041+0000 7feb23fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cdf2291f0 0x564cdf22b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.041+0000 7feb2a59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564cdf224080 0x564cdf228b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.041+0000 7feb2a59c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564cdf224080 0x564cdf228b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.46:41760/0 (socket says 172.21.15.46:41760) 2024-04-26T21:34:41.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.041+0000 7feb2a59c640 1 -- 172.21.15.46:0/2003794307 learned_addr learned my addr 172.21.15.46:0/2003794307 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:41.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.041+0000 7feb2ad9d640 1 --2- 172.21.15.46:0/2003794307 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564cdf223690 0x564cdf223ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.041+0000 7feb2ad9d640 1 --2- 172.21.15.46:0/836376564 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7feb180a3c40 0x7feb180a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.041+0000 7feb237fe640 1 -- 172.21.15.46:0/836376564 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7feb140d02a0 con 0x7feb2c101b60 2024-04-26T21:34:41.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.041+0000 7feb2a59c640 1 -- 172.21.15.46:0/2003794307 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564cdf223690 msgr2=0x564cdf223ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:41.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.041+0000 7feb2a59c640 1 --2- 172.21.15.46:0/2003794307 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564cdf223690 0x564cdf223ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.041+0000 7feb2a59c640 1 -- 172.21.15.46:0/2003794307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cdf2291f0 msgr2=0x564cdf22b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:41.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.041+0000 7feb2a59c640 1 --2- 172.21.15.46:0/2003794307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cdf2291f0 0x564cdf22b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.041+0000 7feb2a59c640 1 -- 172.21.15.46:0/2003794307 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cdf140750 con 0x564cdf224080 2024-04-26T21:34:41.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.041+0000 7feb2ad9d640 1 --2- 172.21.15.46:0/836376564 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7feb180a3c40 0x7feb180a6100 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7feb240098d0 tx=0x7feb240093c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:41.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.041+0000 7feb2a59c640 1 --2- 172.21.15.46:0/2003794307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564cdf224080 0x564cdf228b80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7feb1c00b820 tx=0x7feb1c00bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c00676d050ae1d95 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:41.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.042+0000 7feb20ff9640 1 -- 172.21.15.46:0/2003794307 <== mon.2 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb1c00d950 con 0x564cdf224080 2024-04-26T21:34:41.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.042+0000 7feb20ff9640 1 -- 172.21.15.46:0/2003794307 <== mon.2 v2:172.21.15.50:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb1c019070 con 0x564cdf224080 2024-04-26T21:34:41.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.042+0000 7feb20ff9640 1 -- 172.21.15.46:0/2003794307 <== mon.2 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb1c0123e0 con 0x564cdf224080 2024-04-26T21:34:41.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.042+0000 7feb335f5740 1 -- 172.21.15.46:0/2003794307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564cdf224080 msgr2=0x564cdf228b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.042+0000 7feb335f5740 1 --2- 172.21.15.46:0/2003794307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564cdf224080 0x564cdf228b80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7feb1c00b820 tx=0x7feb1c00bce0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.042+0000 7feb335f5740 1 -- 172.21.15.46:0/2003794307 shutdown_connections 2024-04-26T21:34:41.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.042+0000 7feb335f5740 1 --2- 172.21.15.46:0/2003794307 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cdf2291f0 0x564cdf22b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.042+0000 7feb335f5740 1 --2- 172.21.15.46:0/2003794307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564cdf224080 0x564cdf228b80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.042+0000 7feb335f5740 1 --2- 172.21.15.46:0/2003794307 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564cdf223690 0x564cdf223ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.042+0000 7feb335f5740 1 -- 172.21.15.46:0/2003794307 >> 172.21.15.46:0/2003794307 conn(0x564cdf21ed00 msgr2=0x564cdf21f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:41.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.042+0000 7feb335f5740 1 -- 172.21.15.46:0/2003794307 shutdown_connections 2024-04-26T21:34:41.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.042+0000 7feb335f5740 1 -- 172.21.15.46:0/2003794307 wait complete. 2024-04-26T21:34:41.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.043+0000 7feb335f5740 1 -- start start 2024-04-26T21:34:41.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.043+0000 7feb335f5740 1 Processor -- start 2024-04-26T21:34:41.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.043+0000 7feb335f5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564cdf223690 0x564cdf22db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.043+0000 7feb335f5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cdf224080 0x564cdf22e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.043+0000 7feb335f5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564cdf2291f0 0x564cdf235050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.043+0000 7feb335f5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564cdf22c1a0 con 0x564cdf224080 2024-04-26T21:34:41.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.043+0000 7feb335f5740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x564cdf10ffe0 con 0x564cdf2291f0 2024-04-26T21:34:41.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.043+0000 7feb335f5740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x564cdf22c320 con 0x564cdf223690 2024-04-26T21:34:41.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.043+0000 7feb335f5740 10 client.0 _ll_register_callbacks cb 0x564cdf126150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:34:41.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.043+0000 7feb23fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cdf224080 0x564cdf22e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.043+0000 7feb23fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cdf224080 0x564cdf22e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42044/0 (socket says 172.21.15.46:42044) 2024-04-26T21:34:41.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.043+0000 7feb23fff640 1 -- 172.21.15.46:0/1474231123 learned_addr learned my addr 172.21.15.46:0/1474231123 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:41.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.043+0000 7feb2a59c640 1 --2- 172.21.15.46:0/1474231123 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564cdf223690 0x564cdf22db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.043+0000 7feb2ad9d640 1 --2- 172.21.15.46:0/1474231123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564cdf2291f0 0x564cdf235050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.043+0000 7feb23fff640 1 -- 172.21.15.46:0/1474231123 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564cdf223690 msgr2=0x564cdf22db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:41.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.043+0000 7feb23fff640 1 --2- 172.21.15.46:0/1474231123 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564cdf223690 0x564cdf22db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.043+0000 7feb23fff640 1 -- 172.21.15.46:0/1474231123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564cdf2291f0 msgr2=0x564cdf235050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:41.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.043+0000 7feb23fff640 1 --2- 172.21.15.46:0/1474231123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564cdf2291f0 0x564cdf235050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.043+0000 7feb23fff640 1 -- 172.21.15.46:0/1474231123 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cdee9a2d0 con 0x564cdf224080 2024-04-26T21:34:41.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.044+0000 7feb23fff640 1 --2- 172.21.15.46:0/1474231123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cdf224080 0x564cdf22e080 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7feb1409c150 tx=0x7feb1409c080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a35abf63282db05e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:41.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.044+0000 7feb11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-26T21:34:41.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.044+0000 7feb11ffb640 1 -- 172.21.15.46:0/1474231123 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb140b5060 con 0x564cdf224080 2024-04-26T21:34:41.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.044+0000 7feb11ffb640 1 -- 172.21.15.46:0/1474231123 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7feb1409b8a0 con 0x564cdf224080 2024-04-26T21:34:41.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.044+0000 7feb335f5740 1 -- 172.21.15.46:0/1474231123 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb1c019070 con 0x564cdf224080 2024-04-26T21:34:41.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.044+0000 7feb11ffb640 1 -- 172.21.15.46:0/1474231123 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7feb1409c450 con 0x564cdf224080 2024-04-26T21:34:41.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.044+0000 7feb335f5740 10 client.15240 fetch_fsmap learned FSMap version 22 2024-04-26T21:34:41.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.044+0000 7feb335f5740 1 -- 172.21.15.46:0/1474231123 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x564cdef646b0 con 0x564cdf224080 2024-04-26T21:34:41.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.044+0000 7feb11ffb640 1 -- 172.21.15.46:0/1474231123 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb1c019070 con 0x564cdf224080 2024-04-26T21:34:41.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.044+0000 7feb11ffb640 1 -- 172.21.15.46:0/1474231123 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7feb140bb170 con 0x564cdf224080 2024-04-26T21:34:41.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.044+0000 7feb335f5740 10 client.15240 fetch_fsmap finished waiting for FSMap version 22 2024-04-26T21:34:41.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.044+0000 7feb335f5740 10 client.15240 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T21:34:41.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.044+0000 7feb335f5740 20 client.15240 populate_metadata read hostname 'smithi046' 2024-04-26T21:34:41.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.044+0000 7feb335f5740 1 --2- 172.21.15.46:0/1474231123 >> [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] conn(0x564cdf23a490 0x564cdf23b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.044+0000 7feb335f5740 4 client.15240 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:34:41.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.044+0000 7feb335f5740 1 -- 172.21.15.46:0/1474231123 --> [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564cdf196350 con 0x564cdf23a490 2024-04-26T21:34:41.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.045+0000 7feb2a59c640 1 --2- 172.21.15.46:0/1474231123 >> [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] conn(0x564cdf23a490 0x564cdf23b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.045+0000 7feb11ffb640 10 client.15240 ms_handle_connect on v2:172.21.15.50:6836/3189620395 2024-04-26T21:34:41.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.045+0000 7feb2a59c640 1 --2- 172.21.15.46:0/1474231123 >> [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] conn(0x564cdf23a490 0x564cdf23b900 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=72bd2344f3ba3ee2 server_cookie=c8cb4dcc93978111 in_seq=0 out_seq=0 2024-04-26T21:34:41.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.046+0000 7feb11ffb640 1 -- 172.21.15.46:0/1474231123 <== mds.2 v2:172.21.15.50:6836/3189620395 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564cdf196350 con 0x564cdf23a490 2024-04-26T21:34:41.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.046+0000 7feb11ffb640 10 client.15240 handle_command_reply: tid=0 2024-04-26T21:34:41.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.046+0000 7feb335f5740 1 client.15240 shutdown 2024-04-26T21:34:41.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.046+0000 7feb335f5740 1 -- 172.21.15.46:0/1474231123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cdf224080 msgr2=0x564cdf22e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.046+0000 7feb335f5740 1 --2- 172.21.15.46:0/1474231123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cdf224080 0x564cdf22e080 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7feb1409c150 tx=0x7feb1409c080 comp rx=0 tx=0).stop 2024-04-26T21:34:41.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.046+0000 7feb335f5740 1 -- 172.21.15.46:0/1474231123 shutdown_connections 2024-04-26T21:34:41.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.046+0000 7feb335f5740 1 --2- 172.21.15.46:0/1474231123 >> [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] conn(0x564cdf23a490 0x564cdf23b900 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.046+0000 7feb335f5740 1 --2- 172.21.15.46:0/1474231123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564cdf2291f0 0x564cdf235050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.046+0000 7feb335f5740 1 --2- 172.21.15.46:0/1474231123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cdf224080 0x564cdf22e080 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.046+0000 7feb335f5740 1 --2- 172.21.15.46:0/1474231123 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564cdf223690 0x564cdf22db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.046+0000 7feb11ffb640 0 client.15240 ms_handle_reset on v2:172.21.15.50:6836/3189620395 2024-04-26T21:34:41.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.046+0000 7feb335f5740 1 -- 172.21.15.46:0/1474231123 >> 172.21.15.46:0/1474231123 conn(0x564cdf21ed00 msgr2=0x564cdf21df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:41.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.046+0000 7feb335f5740 1 -- 172.21.15.46:0/1474231123 shutdown_connections 2024-04-26T21:34:41.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.046+0000 7feb335f5740 1 -- 172.21.15.46:0/1474231123 wait complete. 2024-04-26T21:34:41.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.046+0000 7feb335f5740 20 client.15240 trim_cache size 0 max 16384 2024-04-26T21:34:41.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.065+0000 7feb335f5740 1 Processor -- start 2024-04-26T21:34:41.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.065+0000 7feb335f5740 1 -- start start 2024-04-26T21:34:41.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.065+0000 7feb335f5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564cdf223690 0x564cdf243be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.065+0000 7feb335f5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cdf224080 0x564cdf244120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.065+0000 7feb335f5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564cdf2291f0 0x564cdf235010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.065+0000 7feb335f5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564cdf239c40 con 0x564cdf224080 2024-04-26T21:34:41.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.065+0000 7feb335f5740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x564cdf23a310 con 0x564cdf2291f0 2024-04-26T21:34:41.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.065+0000 7feb335f5740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x564cdf23a9e0 con 0x564cdf223690 2024-04-26T21:34:41.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.065+0000 7feb23fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cdf224080 0x564cdf244120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.065+0000 7feb23fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cdf224080 0x564cdf244120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42056/0 (socket says 172.21.15.46:42056) 2024-04-26T21:34:41.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.065+0000 7feb23fff640 1 -- 172.21.15.46:0/2947855351 learned_addr learned my addr 172.21.15.46:0/2947855351 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:41.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.065+0000 7feb2a59c640 1 --2- 172.21.15.46:0/2947855351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564cdf223690 0x564cdf243be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.065+0000 7feb23fff640 1 -- 172.21.15.46:0/2947855351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564cdf223690 msgr2=0x564cdf243be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:41.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.065+0000 7feb23fff640 1 --2- 172.21.15.46:0/2947855351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564cdf223690 0x564cdf243be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.065+0000 7feb2ad9d640 1 --2- 172.21.15.46:0/2947855351 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564cdf2291f0 0x564cdf235010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.065+0000 7feb23fff640 1 -- 172.21.15.46:0/2947855351 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564cdf2291f0 msgr2=0x564cdf235010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:41.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.065+0000 7feb23fff640 1 --2- 172.21.15.46:0/2947855351 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564cdf2291f0 0x564cdf235010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.065+0000 7feb23fff640 1 -- 172.21.15.46:0/2947855351 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cdf21f750 con 0x564cdf224080 2024-04-26T21:34:41.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.065+0000 7feb23fff640 1 --2- 172.21.15.46:0/2947855351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cdf224080 0x564cdf244120 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7feb140ce200 tx=0x7feb140d05f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca11d7c70a969449 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:41.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.066+0000 7feb13fff640 1 -- 172.21.15.46:0/2947855351 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb140b5040 con 0x564cdf224080 2024-04-26T21:34:41.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.066+0000 7feb13fff640 1 -- 172.21.15.46:0/2947855351 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb140a0800 con 0x564cdf224080 2024-04-26T21:34:41.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.066+0000 7feb13fff640 1 -- 172.21.15.46:0/2947855351 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb140983e0 con 0x564cdf224080 2024-04-26T21:34:41.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.066+0000 7feb335f5740 1 -- 172.21.15.46:0/2947855351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cdf224080 msgr2=0x564cdf244120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.066+0000 7feb335f5740 1 --2- 172.21.15.46:0/2947855351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cdf224080 0x564cdf244120 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7feb140ce200 tx=0x7feb140d05f0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.066+0000 7feb335f5740 1 -- 172.21.15.46:0/2947855351 shutdown_connections 2024-04-26T21:34:41.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.066+0000 7feb335f5740 1 --2- 172.21.15.46:0/2947855351 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564cdf2291f0 0x564cdf235010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.066+0000 7feb335f5740 1 --2- 172.21.15.46:0/2947855351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cdf224080 0x564cdf244120 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.066+0000 7feb335f5740 1 --2- 172.21.15.46:0/2947855351 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564cdf223690 0x564cdf243be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.066+0000 7feb335f5740 1 -- 172.21.15.46:0/2947855351 >> 172.21.15.46:0/2947855351 conn(0x564cdf21ed00 msgr2=0x564cdf22db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:41.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.066+0000 7feb335f5740 1 -- 172.21.15.46:0/2947855351 shutdown_connections 2024-04-26T21:34:41.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.066+0000 7feb335f5740 1 -- 172.21.15.46:0/2947855351 wait complete. 2024-04-26T21:34:41.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.066+0000 7feb335f5740 1 -- start start 2024-04-26T21:34:41.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.067+0000 7feb335f5740 1 Processor -- start 2024-04-26T21:34:41.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.067+0000 7feb335f5740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564cdf223690 0x564cdf2348e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.067+0000 7feb335f5740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564cdf224080 0x564cdf234e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.067+0000 7feb335f5740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cdf2291f0 0x564cdf231380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.067+0000 7feb335f5740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x564cdf23b780 con 0x564cdf2291f0 2024-04-26T21:34:41.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.067+0000 7feb335f5740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x564cdf23b0b0 con 0x564cdf224080 2024-04-26T21:34:41.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.067+0000 7feb335f5740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x564cdf23be50 con 0x564cdf223690 2024-04-26T21:34:41.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.067+0000 7feb335f5740 10 client.0 _ll_register_callbacks cb 0x7feb140980b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:34:41.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.067+0000 7feb2ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cdf2291f0 0x564cdf231380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.067+0000 7feb2ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cdf2291f0 0x564cdf231380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42058/0 (socket says 172.21.15.46:42058) 2024-04-26T21:34:41.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.067+0000 7feb2ad9d640 1 -- 172.21.15.46:0/1595743210 learned_addr learned my addr 172.21.15.46:0/1595743210 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:41.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.067+0000 7feb23fff640 1 --2- 172.21.15.46:0/1595743210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564cdf224080 0x564cdf234e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.067+0000 7feb2a59c640 1 --2- 172.21.15.46:0/1595743210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564cdf223690 0x564cdf2348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.067+0000 7feb2ad9d640 1 -- 172.21.15.46:0/1595743210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564cdf223690 msgr2=0x564cdf2348e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:41.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.067+0000 7feb2ad9d640 1 --2- 172.21.15.46:0/1595743210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564cdf223690 0x564cdf2348e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.067+0000 7feb2ad9d640 1 -- 172.21.15.46:0/1595743210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564cdf224080 msgr2=0x564cdf234e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:41.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.067+0000 7feb2ad9d640 1 --2- 172.21.15.46:0/1595743210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564cdf224080 0x564cdf234e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.067+0000 7feb2ad9d640 1 -- 172.21.15.46:0/1595743210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cdf231ac0 con 0x564cdf2291f0 2024-04-26T21:34:41.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.067+0000 7feb2a59c640 1 --2- 172.21.15.46:0/1595743210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564cdf223690 0x564cdf2348e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:34:41.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.067+0000 7feb2ad9d640 1 --2- 172.21.15.46:0/1595743210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cdf2291f0 0x564cdf231380 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7feb24009ca0 tx=0x7feb240095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d4d846aa32c6e3c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:41.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.067+0000 7feb11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-26T21:34:41.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.067+0000 7feb11ffb640 1 -- 172.21.15.46:0/1595743210 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2405a060 con 0x564cdf2291f0 2024-04-26T21:34:41.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.068+0000 7feb11ffb640 1 -- 172.21.15.46:0/1595743210 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7feb24004310 con 0x564cdf2291f0 2024-04-26T21:34:41.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.068+0000 7feb335f5740 1 -- 172.21.15.46:0/1595743210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb140a0800 con 0x564cdf2291f0 2024-04-26T21:34:41.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.068+0000 7feb11ffb640 1 -- 172.21.15.46:0/1595743210 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7feb24004510 con 0x564cdf2291f0 2024-04-26T21:34:41.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.068+0000 7feb335f5740 10 client.15246 fetch_fsmap learned FSMap version 22 2024-04-26T21:34:41.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.068+0000 7feb335f5740 1 -- 172.21.15.46:0/1595743210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x564cdf2a10c0 con 0x564cdf2291f0 2024-04-26T21:34:41.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.068+0000 7feb11ffb640 1 -- 172.21.15.46:0/1595743210 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb140a0800 con 0x564cdf2291f0 2024-04-26T21:34:41.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.068+0000 7feb11ffb640 1 -- 172.21.15.46:0/1595743210 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7feb24054070 con 0x564cdf2291f0 2024-04-26T21:34:41.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.068+0000 7feb335f5740 10 client.15246 fetch_fsmap finished waiting for FSMap version 22 2024-04-26T21:34:41.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.068+0000 7feb335f5740 10 client.15246 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T21:34:41.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.068+0000 7feb335f5740 20 client.15246 populate_metadata read hostname 'smithi046' 2024-04-26T21:34:41.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.068+0000 7feb335f5740 1 --2- 172.21.15.46:0/1595743210 >> [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] conn(0x564cdf2a5d20 0x564cdf2a2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.068+0000 7feb335f5740 4 client.15246 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T21:34:41.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.068+0000 7feb335f5740 1 -- 172.21.15.46:0/1595743210 --> [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] -- command(tid 0: {"prefix": "status"}) -- 0x564cdf1901c0 con 0x564cdf2a5d20 2024-04-26T21:34:41.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.068+0000 7feb2a59c640 1 --2- 172.21.15.46:0/1595743210 >> [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] conn(0x564cdf2a5d20 0x564cdf2a2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.069+0000 7feb2a59c640 1 --2- 172.21.15.46:0/1595743210 >> [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] conn(0x564cdf2a5d20 0x564cdf2a2160 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b4d5f16540617e86 server_cookie=5518d4bfce0d1d63 in_seq=0 out_seq=0 2024-04-26T21:34:41.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.069+0000 7feb11ffb640 10 client.15246 ms_handle_connect on v2:172.21.15.50:6836/3189620395 2024-04-26T21:34:41.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.069+0000 7feb11ffb640 1 -- 172.21.15.46:0/1595743210 <== mds.2 v2:172.21.15.50:6836/3189620395 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x564cdf1901c0 con 0x564cdf2a5d20 2024-04-26T21:34:41.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.069+0000 7feb11ffb640 10 client.15246 handle_command_reply: tid=0 2024-04-26T21:34:41.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.069+0000 7feb335f5740 1 client.15246 shutdown 2024-04-26T21:34:41.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.069+0000 7feb335f5740 1 -- 172.21.15.46:0/1595743210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cdf2291f0 msgr2=0x564cdf231380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.069+0000 7feb335f5740 1 --2- 172.21.15.46:0/1595743210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cdf2291f0 0x564cdf231380 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7feb24009ca0 tx=0x7feb240095c0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.069+0000 7feb335f5740 1 -- 172.21.15.46:0/1595743210 shutdown_connections 2024-04-26T21:34:41.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.069+0000 7feb335f5740 1 --2- 172.21.15.46:0/1595743210 >> [v2:172.21.15.50:6836/3189620395,v1:172.21.15.50:6837/3189620395] conn(0x564cdf2a5d20 0x564cdf2a2160 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.069+0000 7feb335f5740 1 --2- 172.21.15.46:0/1595743210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564cdf2291f0 0x564cdf231380 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.069+0000 7feb335f5740 1 --2- 172.21.15.46:0/1595743210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x564cdf224080 0x564cdf234e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.069+0000 7feb11ffb640 0 client.15246 ms_handle_reset on v2:172.21.15.50:6836/3189620395 2024-04-26T21:34:41.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.069+0000 7feb335f5740 1 --2- 172.21.15.46:0/1595743210 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x564cdf223690 0x564cdf2348e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.069+0000 7feb335f5740 1 -- 172.21.15.46:0/1595743210 >> 172.21.15.46:0/1595743210 conn(0x564cdf21ed00 msgr2=0x564cdf221d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:41.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.069+0000 7feb335f5740 1 -- 172.21.15.46:0/1595743210 shutdown_connections 2024-04-26T21:34:41.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.069+0000 7feb335f5740 1 -- 172.21.15.46:0/1595743210 wait complete. 2024-04-26T21:34:41.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.069+0000 7feb335f5740 20 client.15246 trim_cache size 0 max 16384 2024-04-26T21:34:41.078 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-26T21:34:41.078 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36c7013a-0413-11ef-bc93-c7b262605968", 2024-04-26T21:34:41.078 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 2, 2024-04-26T21:34:41.079 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 24509, 2024-04-26T21:34:41.079 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-04-26T21:34:41.079 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-04-26T21:34:41.079 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-04-26T21:34:41.079 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 112.186272071, 2024-04-26T21:34:41.079 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 22, 2024-04-26T21:34:41.079 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 87, 2024-04-26T21:34:41.079 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 85, 2024-04-26T21:34:41.079 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 133.75291911299999 2024-04-26T21:34:41.079 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-26T21:34:41.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.070+0000 7feb30808640 1 -- 172.21.15.46:0/836376564 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7feb180a3c40 msgr2=0x7feb180a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.070+0000 7feb30808640 1 --2- 172.21.15.46:0/836376564 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7feb180a3c40 0x7feb180a6100 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7feb240098d0 tx=0x7feb240093c0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.070+0000 7feb30808640 1 -- 172.21.15.46:0/836376564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb2c101b60 msgr2=0x7feb2c19b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.070+0000 7feb30808640 1 --2- 172.21.15.46:0/836376564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb2c101b60 0x7feb2c19b340 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7feb1400c890 tx=0x7feb1400cd50 comp rx=0 tx=0).stop 2024-04-26T21:34:41.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.070+0000 7feb30808640 1 -- 172.21.15.46:0/836376564 shutdown_connections 2024-04-26T21:34:41.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.070+0000 7feb30808640 1 --2- 172.21.15.46:0/836376564 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7feb180a3c40 0x7feb180a6100 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.070+0000 7feb30808640 1 --2- 172.21.15.46:0/836376564 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7feb2c102500 0x7feb2c1a23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.070+0000 7feb30808640 1 --2- 172.21.15.46:0/836376564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb2c101b60 0x7feb2c19b340 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.070+0000 7feb30808640 1 --2- 172.21.15.46:0/836376564 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7feb2c101200 0x7feb2c19ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.070+0000 7feb30808640 1 -- 172.21.15.46:0/836376564 >> 172.21.15.46:0/836376564 conn(0x7feb2c0fce20 msgr2=0x7feb2c0fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:41.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.070+0000 7feb30808640 1 -- 172.21.15.46:0/836376564 shutdown_connections 2024-04-26T21:34:41.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.070+0000 7feb30808640 1 -- 172.21.15.46:0/836376564 wait complete. 2024-04-26T21:34:41.082 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-26T21:34:41.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.183+0000 7f4e1bfda640 1 -- 172.21.15.46:0/2463464497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e14105d40 msgr2=0x7f4e14108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.183+0000 7f4e1bfda640 1 --2- 172.21.15.46:0/2463464497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e14105d40 0x7f4e14108130 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f4e04009900 tx=0x7f4e0402a410 comp rx=0 tx=0).stop 2024-04-26T21:34:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.183+0000 7f4e1bfda640 1 -- 172.21.15.46:0/2463464497 shutdown_connections 2024-04-26T21:34:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.183+0000 7f4e1bfda640 1 --2- 172.21.15.46:0/2463464497 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e14108670 0x7f4e1410aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.183+0000 7f4e1bfda640 1 --2- 172.21.15.46:0/2463464497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e14105d40 0x7f4e14108130 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.183+0000 7f4e1bfda640 1 --2- 172.21.15.46:0/2463464497 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e14103410 0x7f4e14105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.183+0000 7f4e1bfda640 1 -- 172.21.15.46:0/2463464497 >> 172.21.15.46:0/2463464497 conn(0x7f4e140fce20 msgr2=0x7f4e140ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.183+0000 7f4e1bfda640 1 -- 172.21.15.46:0/2463464497 shutdown_connections 2024-04-26T21:34:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.183+0000 7f4e1bfda640 1 -- 172.21.15.46:0/2463464497 wait complete. 2024-04-26T21:34:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.184+0000 7f4e1bfda640 1 Processor -- start 2024-04-26T21:34:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.184+0000 7f4e1bfda640 1 -- start start 2024-04-26T21:34:41.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.184+0000 7f4e1bfda640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e14103410 0x7f4e14198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.184+0000 7f4e1bfda640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e14105d40 0x7f4e141991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.184+0000 7f4e1bfda640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e14108670 0x7f4e141a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.184+0000 7f4e1bfda640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4e1410e4c0 con 0x7f4e14105d40 2024-04-26T21:34:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.184+0000 7f4e1bfda640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4e1410e340 con 0x7f4e14103410 2024-04-26T21:34:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.184+0000 7f4e1bfda640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4e1410e640 con 0x7f4e14108670 2024-04-26T21:34:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.184+0000 7f4e19575640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e14105d40 0x7f4e141991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.184+0000 7f4e19575640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e14105d40 0x7f4e141991c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42074/0 (socket says 172.21.15.46:42074) 2024-04-26T21:34:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.184+0000 7f4e19575640 1 -- 172.21.15.46:0/1513749409 learned_addr learned my addr 172.21.15.46:0/1513749409 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.184+0000 7f4e19d76640 1 --2- 172.21.15.46:0/1513749409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e14103410 0x7f4e14198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.184+0000 7f4e1a577640 1 --2- 172.21.15.46:0/1513749409 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e14108670 0x7f4e141a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.184+0000 7f4e19575640 1 -- 172.21.15.46:0/1513749409 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e14108670 msgr2=0x7f4e141a01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.184+0000 7f4e19575640 1 --2- 172.21.15.46:0/1513749409 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e14108670 0x7f4e141a01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.184+0000 7f4e19575640 1 -- 172.21.15.46:0/1513749409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e14103410 msgr2=0x7f4e14198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.184+0000 7f4e19575640 1 --2- 172.21.15.46:0/1513749409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e14103410 0x7f4e14198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.184+0000 7f4e19575640 1 -- 172.21.15.46:0/1513749409 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e141a0840 con 0x7f4e14105d40 2024-04-26T21:34:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.185+0000 7f4e19575640 1 --2- 172.21.15.46:0/1513749409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e14105d40 0x7f4e141991c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f4e04006010 tx=0x7f4e040042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.185+0000 7f4e0affd640 1 -- 172.21.15.46:0/1513749409 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0403f030 con 0x7f4e14105d40 2024-04-26T21:34:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.185+0000 7f4e1bfda640 1 -- 172.21.15.46:0/1513749409 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e141a0ad0 con 0x7f4e14105d40 2024-04-26T21:34:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.185+0000 7f4e0affd640 1 -- 172.21.15.46:0/1513749409 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e040318f0 con 0x7f4e14105d40 2024-04-26T21:34:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.185+0000 7f4e0affd640 1 -- 172.21.15.46:0/1513749409 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e04035070 con 0x7f4e14105d40 2024-04-26T21:34:41.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.185+0000 7f4e1bfda640 1 -- 172.21.15.46:0/1513749409 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e141a1010 con 0x7f4e14105d40 2024-04-26T21:34:41.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.187+0000 7f4e0affd640 1 -- 172.21.15.46:0/1513749409 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4e04042030 con 0x7f4e14105d40 2024-04-26T21:34:41.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.187+0000 7f4e1edcc740 1 Processor -- start 2024-04-26T21:34:41.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.187+0000 7f4e1edcc740 1 -- start start 2024-04-26T21:34:41.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.188+0000 7f4e1edcc740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561b3e760690 0x561b3e760ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.188+0000 7f4e1edcc740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e761080 0x561b3e765b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.188+0000 7f4e1edcc740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b3e7661f0 0x561b3e768680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.188+0000 7f4e1edcc740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561b3e6c5740 con 0x561b3e761080 2024-04-26T21:34:41.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.188+0000 7f4e1edcc740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x561b3e6613a0 con 0x561b3e7661f0 2024-04-26T21:34:41.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.188+0000 7f4e1edcc740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x561b3e6c1700 con 0x561b3e760690 2024-04-26T21:34:41.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.188+0000 7f4e0affd640 1 --2- 172.21.15.46:0/1513749409 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4e000a3c40 0x7f4e000a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.188+0000 7f4e19d76640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e761080 0x561b3e765b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.188+0000 7f4e1a577640 1 --2- 172.21.15.46:0/1513749409 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4e000a3c40 0x7f4e000a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.188+0000 7f4e19d76640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e761080 0x561b3e765b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42082/0 (socket says 172.21.15.46:42082) 2024-04-26T21:34:41.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.188+0000 7f4e19d76640 1 -- 172.21.15.46:0/1592512552 learned_addr learned my addr 172.21.15.46:0/1592512552 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:41.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.188+0000 7f4e0affd640 1 -- 172.21.15.46:0/1513749409 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f4e040f0c10 con 0x7f4e14105d40 2024-04-26T21:34:41.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.188+0000 7f4e19575640 1 --2- 172.21.15.46:0/1592512552 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b3e7661f0 0x561b3e768680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.188+0000 7f4e1a577640 1 --2- 172.21.15.46:0/1592512552 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561b3e760690 0x561b3e760ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.188+0000 7f4e19d76640 1 -- 172.21.15.46:0/1592512552 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561b3e760690 msgr2=0x561b3e760ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.188+0000 7f4e19d76640 1 --2- 172.21.15.46:0/1592512552 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561b3e760690 0x561b3e760ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.188+0000 7f4e19d76640 1 -- 172.21.15.46:0/1592512552 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b3e7661f0 msgr2=0x561b3e768680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.188+0000 7f4e19d76640 1 --2- 172.21.15.46:0/1592512552 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b3e7661f0 0x561b3e768680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.188+0000 7f4e19d76640 1 -- 172.21.15.46:0/1592512552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b3e67d750 con 0x561b3e761080 2024-04-26T21:34:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.188+0000 7f4e1a577640 1 --2- 172.21.15.46:0/1513749409 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4e000a3c40 0x7f4e000a6100 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c009270 tx=0x7f4e0c012c10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.188+0000 7f4e19d76640 1 --2- 172.21.15.46:0/1592512552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e761080 0x561b3e765b80 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f4e1000bc90 tx=0x7f4e1000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e34277b4b47bb2ec server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.188+0000 7f4dfffff640 1 -- 172.21.15.46:0/1592512552 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e10015070 con 0x561b3e761080 2024-04-26T21:34:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.189+0000 7f4dfffff640 1 -- 172.21.15.46:0/1592512552 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e1001a910 con 0x561b3e761080 2024-04-26T21:34:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.189+0000 7f4dfffff640 1 -- 172.21.15.46:0/1592512552 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e100118a0 con 0x561b3e761080 2024-04-26T21:34:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.189+0000 7f4e1edcc740 1 -- 172.21.15.46:0/1592512552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e761080 msgr2=0x561b3e765b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.189+0000 7f4e1edcc740 1 --2- 172.21.15.46:0/1592512552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e761080 0x561b3e765b80 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f4e1000bc90 tx=0x7f4e1000f730 comp rx=0 tx=0).stop 2024-04-26T21:34:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.189+0000 7f4e1edcc740 1 -- 172.21.15.46:0/1592512552 shutdown_connections 2024-04-26T21:34:41.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.189+0000 7f4e1edcc740 1 --2- 172.21.15.46:0/1592512552 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b3e7661f0 0x561b3e768680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.189+0000 7f4e1edcc740 1 --2- 172.21.15.46:0/1592512552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e761080 0x561b3e765b80 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.189+0000 7f4e1edcc740 1 --2- 172.21.15.46:0/1592512552 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561b3e760690 0x561b3e760ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.189+0000 7f4e1edcc740 1 -- 172.21.15.46:0/1592512552 >> 172.21.15.46:0/1592512552 conn(0x561b3e75bd00 msgr2=0x561b3e75c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.189+0000 7f4e1edcc740 1 -- 172.21.15.46:0/1592512552 shutdown_connections 2024-04-26T21:34:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.189+0000 7f4e1edcc740 1 -- 172.21.15.46:0/1592512552 wait complete. 2024-04-26T21:34:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.190+0000 7f4e1edcc740 1 -- start start 2024-04-26T21:34:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.190+0000 7f4e1edcc740 1 Processor -- start 2024-04-26T21:34:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.190+0000 7f4e1edcc740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e760690 0x561b3e76ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.190+0000 7f4e1edcc740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561b3e761080 0x561b3e76b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.190+0000 7f4e1edcc740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b3e7661f0 0x561b3e772090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.190+0000 7f4e1edcc740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561b3e7691a0 con 0x561b3e760690 2024-04-26T21:34:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.190+0000 7f4e1edcc740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x561b3e64cfe0 con 0x561b3e7661f0 2024-04-26T21:34:41.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.190+0000 7f4e1edcc740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x561b3e769320 con 0x561b3e761080 2024-04-26T21:34:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.190+0000 7f4e19d76640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e760690 0x561b3e76ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.190+0000 7f4e1edcc740 10 client.0 _ll_register_callbacks cb 0x561b3e663150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:34:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.190+0000 7f4e19d76640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e760690 0x561b3e76ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42086/0 (socket says 172.21.15.46:42086) 2024-04-26T21:34:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.190+0000 7f4e19d76640 1 -- 172.21.15.46:0/3718373628 learned_addr learned my addr 172.21.15.46:0/3718373628 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.190+0000 7f4e1a577640 1 --2- 172.21.15.46:0/3718373628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b3e7661f0 0x561b3e772090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.190+0000 7f4e19d76640 1 -- 172.21.15.46:0/3718373628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561b3e761080 msgr2=0x561b3e76b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.190+0000 7f4e19d76640 1 --2- 172.21.15.46:0/3718373628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561b3e761080 0x561b3e76b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.190+0000 7f4e19d76640 1 -- 172.21.15.46:0/3718373628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b3e7661f0 msgr2=0x561b3e772090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.190+0000 7f4e19d76640 1 --2- 172.21.15.46:0/3718373628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b3e7661f0 0x561b3e772090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.191+0000 7f4e19d76640 1 -- 172.21.15.46:0/3718373628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b3e3d72d0 con 0x561b3e760690 2024-04-26T21:34:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.191+0000 7f4e19d76640 1 --2- 172.21.15.46:0/3718373628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e760690 0x561b3e76ab10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4e100058c0 tx=0x7f4e1001a690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81d1a3a1cd5741e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.191+0000 7f4dfd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-26T21:34:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.191+0000 7f4dfd7fa640 1 -- 172.21.15.46:0/3718373628 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e10015070 con 0x561b3e760690 2024-04-26T21:34:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.191+0000 7f4dfd7fa640 1 -- 172.21.15.46:0/3718373628 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f4e1001acd0 con 0x561b3e760690 2024-04-26T21:34:41.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.191+0000 7f4e1edcc740 1 -- 172.21.15.46:0/3718373628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e1001a910 con 0x561b3e760690 2024-04-26T21:34:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.191+0000 7f4dfd7fa640 1 -- 172.21.15.46:0/3718373628 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f4e1002b070 con 0x561b3e760690 2024-04-26T21:34:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.191+0000 7f4e1edcc740 10 client.15258 fetch_fsmap learned FSMap version 22 2024-04-26T21:34:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.191+0000 7f4e1edcc740 1 -- 172.21.15.46:0/3718373628 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x561b3e4a16b0 con 0x561b3e760690 2024-04-26T21:34:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.191+0000 7f4dfd7fa640 1 -- 172.21.15.46:0/3718373628 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e1001a910 con 0x561b3e760690 2024-04-26T21:34:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.191+0000 7f4dfd7fa640 1 -- 172.21.15.46:0/3718373628 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f4e10004a10 con 0x561b3e760690 2024-04-26T21:34:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.191+0000 7f4e1edcc740 10 client.15258 fetch_fsmap finished waiting for FSMap version 22 2024-04-26T21:34:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.191+0000 7f4e1edcc740 10 client.15258 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T21:34:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.191+0000 7f4e1edcc740 20 client.15258 populate_metadata read hostname 'smithi046' 2024-04-26T21:34:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.191+0000 7f4e1edcc740 1 --2- 172.21.15.46:0/3718373628 >> [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] conn(0x561b3e7771c0 0x561b3e778630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.191+0000 7f4e1edcc740 4 client.15258 mds_command: new command op to 24526 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:34:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.191+0000 7f4e1edcc740 1 -- 172.21.15.46:0/3718373628 --> [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b3e6d3350 con 0x561b3e7771c0 2024-04-26T21:34:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.192+0000 7f4e19575640 1 --2- 172.21.15.46:0/3718373628 >> [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] conn(0x561b3e7771c0 0x561b3e778630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.192+0000 7f4e19575640 1 --2- 172.21.15.46:0/3718373628 >> [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] conn(0x561b3e7771c0 0x561b3e778630 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8aa3041a7e3b18e9 server_cookie=6aad7d091d237545 in_seq=0 out_seq=0 2024-04-26T21:34:41.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.192+0000 7f4dfd7fa640 10 client.15258 ms_handle_connect on v2:172.21.15.142:6838/3089509101 2024-04-26T21:34:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.193+0000 7f4dfd7fa640 1 -- 172.21.15.46:0/3718373628 <== mds.1 v2:172.21.15.142:6838/3089509101 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561b3e6d3350 con 0x561b3e7771c0 2024-04-26T21:34:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.193+0000 7f4dfd7fa640 10 client.15258 handle_command_reply: tid=0 2024-04-26T21:34:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.193+0000 7f4e1edcc740 1 client.15258 shutdown 2024-04-26T21:34:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.193+0000 7f4e1edcc740 1 -- 172.21.15.46:0/3718373628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e760690 msgr2=0x561b3e76ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.193+0000 7f4e1edcc740 1 --2- 172.21.15.46:0/3718373628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e760690 0x561b3e76ab10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4e100058c0 tx=0x7f4e1001a690 comp rx=0 tx=0).stop 2024-04-26T21:34:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.193+0000 7f4e1edcc740 1 -- 172.21.15.46:0/3718373628 shutdown_connections 2024-04-26T21:34:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.193+0000 7f4e1edcc740 1 --2- 172.21.15.46:0/3718373628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b3e7661f0 0x561b3e772090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.193+0000 7f4e1edcc740 1 --2- 172.21.15.46:0/3718373628 >> [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] conn(0x561b3e7771c0 0x561b3e778630 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.193+0000 7f4e1edcc740 1 --2- 172.21.15.46:0/3718373628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561b3e761080 0x561b3e76b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.193+0000 7f4dfd7fa640 0 client.15258 ms_handle_reset on v2:172.21.15.142:6838/3089509101 2024-04-26T21:34:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.193+0000 7f4e1edcc740 1 --2- 172.21.15.46:0/3718373628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e760690 0x561b3e76ab10 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.193+0000 7f4e1edcc740 1 -- 172.21.15.46:0/3718373628 >> 172.21.15.46:0/3718373628 conn(0x561b3e75bd00 msgr2=0x561b3e75af00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.193+0000 7f4e1edcc740 1 -- 172.21.15.46:0/3718373628 shutdown_connections 2024-04-26T21:34:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.193+0000 7f4e1edcc740 1 -- 172.21.15.46:0/3718373628 wait complete. 2024-04-26T21:34:41.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.193+0000 7f4e1edcc740 20 client.15258 trim_cache size 0 max 16384 2024-04-26T21:34:41.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.212+0000 7f4e1edcc740 1 Processor -- start 2024-04-26T21:34:41.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.212+0000 7f4e1edcc740 1 -- start start 2024-04-26T21:34:41.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.212+0000 7f4e1edcc740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561b3e760690 0x561b3e75c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.212+0000 7f4e1edcc740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b3e761080 0x561b3e772050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.212+0000 7f4e1edcc740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e7661f0 0x561b3e77f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.212+0000 7f4e1edcc740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561b3e776970 con 0x561b3e7661f0 2024-04-26T21:34:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.212+0000 7f4e1edcc740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x561b3e777040 con 0x561b3e761080 2024-04-26T21:34:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.212+0000 7f4e1edcc740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x561b3e777710 con 0x561b3e760690 2024-04-26T21:34:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.212+0000 7f4e1a577640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e7661f0 0x561b3e77f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.212+0000 7f4e1a577640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e7661f0 0x561b3e77f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42088/0 (socket says 172.21.15.46:42088) 2024-04-26T21:34:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.212+0000 7f4e19575640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b3e761080 0x561b3e772050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.212+0000 7f4e1a577640 1 -- 172.21.15.46:0/390008915 learned_addr learned my addr 172.21.15.46:0/390008915 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.212+0000 7f4e1a577640 1 -- 172.21.15.46:0/390008915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561b3e760690 msgr2=0x561b3e75c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.212+0000 7f4e1a577640 1 --2- 172.21.15.46:0/390008915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561b3e760690 0x561b3e75c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.212+0000 7f4e1a577640 1 -- 172.21.15.46:0/390008915 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b3e761080 msgr2=0x561b3e772050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.212+0000 7f4e1a577640 1 --2- 172.21.15.46:0/390008915 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b3e761080 0x561b3e772050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.212+0000 7f4e1a577640 1 -- 172.21.15.46:0/390008915 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b3e75b840 con 0x561b3e7661f0 2024-04-26T21:34:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.212+0000 7f4e1a577640 1 --2- 172.21.15.46:0/390008915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e7661f0 0x561b3e77f900 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c003cf0 tx=0x7f4e0c01e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d0eb0296a9919f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.212+0000 7f4dff7fe640 1 -- 172.21.15.46:0/390008915 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0c012e60 con 0x561b3e7661f0 2024-04-26T21:34:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.213+0000 7f4dff7fe640 1 -- 172.21.15.46:0/390008915 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e0c024070 con 0x561b3e7661f0 2024-04-26T21:34:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.213+0000 7f4dff7fe640 1 -- 172.21.15.46:0/390008915 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0c0298d0 con 0x561b3e7661f0 2024-04-26T21:34:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.213+0000 7f4e1edcc740 1 -- 172.21.15.46:0/390008915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e7661f0 msgr2=0x561b3e77f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.213+0000 7f4e1edcc740 1 --2- 172.21.15.46:0/390008915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e7661f0 0x561b3e77f900 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c003cf0 tx=0x7f4e0c01e000 comp rx=0 tx=0).stop 2024-04-26T21:34:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.213+0000 7f4e1edcc740 1 -- 172.21.15.46:0/390008915 shutdown_connections 2024-04-26T21:34:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.213+0000 7f4e1edcc740 1 --2- 172.21.15.46:0/390008915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e7661f0 0x561b3e77f900 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.213+0000 7f4e1edcc740 1 --2- 172.21.15.46:0/390008915 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b3e761080 0x561b3e772050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.213+0000 7f4e1edcc740 1 --2- 172.21.15.46:0/390008915 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561b3e760690 0x561b3e75c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.213+0000 7f4e1edcc740 1 -- 172.21.15.46:0/390008915 >> 172.21.15.46:0/390008915 conn(0x561b3e75bd00 msgr2=0x561b3e75f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.213+0000 7f4e1edcc740 1 -- 172.21.15.46:0/390008915 shutdown_connections 2024-04-26T21:34:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.213+0000 7f4e1edcc740 1 -- 172.21.15.46:0/390008915 wait complete. 2024-04-26T21:34:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.213+0000 7f4e1edcc740 1 -- start start 2024-04-26T21:34:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.213+0000 7f4e1edcc740 1 Processor -- start 2024-04-26T21:34:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.214+0000 7f4e1edcc740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e760690 0x561b3e771d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.214+0000 7f4e1edcc740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561b3e761080 0x561b3e76e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.214+0000 7f4e1edcc740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b3e7661f0 0x561b3e76ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.214+0000 7f4e1edcc740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x561b3e7784b0 con 0x561b3e760690 2024-04-26T21:34:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.214+0000 7f4e1edcc740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x561b3e777de0 con 0x561b3e7661f0 2024-04-26T21:34:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.214+0000 7f4e1edcc740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x561b3e778b80 con 0x561b3e761080 2024-04-26T21:34:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.214+0000 7f4e1edcc740 10 client.0 _ll_register_callbacks cb 0x7f4e04005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:34:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.214+0000 7f4e19d76640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e760690 0x561b3e771d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.214+0000 7f4e19d76640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e760690 0x561b3e771d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42100/0 (socket says 172.21.15.46:42100) 2024-04-26T21:34:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.214+0000 7f4e19d76640 1 -- 172.21.15.46:0/2789658183 learned_addr learned my addr 172.21.15.46:0/2789658183 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.214+0000 7f4e1a577640 1 --2- 172.21.15.46:0/2789658183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b3e7661f0 0x561b3e76ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.214+0000 7f4e19d76640 1 -- 172.21.15.46:0/2789658183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561b3e761080 msgr2=0x561b3e76e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.214+0000 7f4e19d76640 1 --2- 172.21.15.46:0/2789658183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561b3e761080 0x561b3e76e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.214+0000 7f4e19d76640 1 -- 172.21.15.46:0/2789658183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b3e7661f0 msgr2=0x561b3e76ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.214+0000 7f4e19d76640 1 --2- 172.21.15.46:0/2789658183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b3e7661f0 0x561b3e76ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.214+0000 7f4e19d76640 1 -- 172.21.15.46:0/2789658183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b3e75ca70 con 0x561b3e760690 2024-04-26T21:34:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.214+0000 7f4e19d76640 1 --2- 172.21.15.46:0/2789658183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e760690 0x561b3e771d50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f4e10004e30 tx=0x7f4e100118b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d57b4fd3cf81fac0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.214+0000 7f4dfd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-26T21:34:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.215+0000 7f4dfd7fa640 1 -- 172.21.15.46:0/2789658183 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e10015070 con 0x561b3e760690 2024-04-26T21:34:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.215+0000 7f4dfd7fa640 1 -- 172.21.15.46:0/2789658183 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f4e1001a880 con 0x561b3e760690 2024-04-26T21:34:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.215+0000 7f4e1edcc740 1 -- 172.21.15.46:0/2789658183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e0c024070 con 0x561b3e760690 2024-04-26T21:34:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.215+0000 7f4dfd7fa640 1 -- 172.21.15.46:0/2789658183 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f4e10011440 con 0x561b3e760690 2024-04-26T21:34:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.215+0000 7f4e1edcc740 10 client.15264 fetch_fsmap learned FSMap version 22 2024-04-26T21:34:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.215+0000 7f4e1edcc740 1 -- 172.21.15.46:0/2789658183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x561b3e7de000 con 0x561b3e760690 2024-04-26T21:34:41.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.215+0000 7f4dfd7fa640 1 -- 172.21.15.46:0/2789658183 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e0c024070 con 0x561b3e760690 2024-04-26T21:34:41.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.215+0000 7f4dfd7fa640 1 -- 172.21.15.46:0/2789658183 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7f4e10004a50 con 0x561b3e760690 2024-04-26T21:34:41.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.215+0000 7f4e1edcc740 10 client.15264 fetch_fsmap finished waiting for FSMap version 22 2024-04-26T21:34:41.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.215+0000 7f4e1edcc740 10 client.15264 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-26T21:34:41.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.215+0000 7f4e1edcc740 20 client.15264 populate_metadata read hostname 'smithi046' 2024-04-26T21:34:41.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.215+0000 7f4e1edcc740 1 --2- 172.21.15.46:0/2789658183 >> [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] conn(0x561b3e7e2c30 0x561b3e7df070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.215+0000 7f4e1edcc740 4 client.15264 mds_command: new command op to 24526 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T21:34:41.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.215+0000 7f4e1edcc740 1 -- 172.21.15.46:0/2789658183 --> [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] -- command(tid 0: {"prefix": "status"}) -- 0x561b3e6cd1c0 con 0x561b3e7e2c30 2024-04-26T21:34:41.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.215+0000 7f4e19575640 1 --2- 172.21.15.46:0/2789658183 >> [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] conn(0x561b3e7e2c30 0x561b3e7df070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.216+0000 7f4dfd7fa640 10 client.15264 ms_handle_connect on v2:172.21.15.142:6838/3089509101 2024-04-26T21:34:41.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.216+0000 7f4e19575640 1 --2- 172.21.15.46:0/2789658183 >> [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] conn(0x561b3e7e2c30 0x561b3e7df070 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=55e5e85dfa5ea3e1 server_cookie=e66a9f2b2a7189 in_seq=0 out_seq=0 2024-04-26T21:34:41.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.216+0000 7f4dfd7fa640 1 -- 172.21.15.46:0/2789658183 <== mds.1 v2:172.21.15.142:6838/3089509101 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x561b3e6cd1c0 con 0x561b3e7e2c30 2024-04-26T21:34:41.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.216+0000 7f4dfd7fa640 10 client.15264 handle_command_reply: tid=0 2024-04-26T21:34:41.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.216+0000 7f4e1edcc740 1 client.15264 shutdown 2024-04-26T21:34:41.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.216+0000 7f4e1edcc740 1 -- 172.21.15.46:0/2789658183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e760690 msgr2=0x561b3e771d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.216+0000 7f4e1edcc740 1 --2- 172.21.15.46:0/2789658183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e760690 0x561b3e771d50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f4e10004e30 tx=0x7f4e100118b0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.216+0000 7f4e1edcc740 1 -- 172.21.15.46:0/2789658183 shutdown_connections 2024-04-26T21:34:41.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.216+0000 7f4e1edcc740 1 --2- 172.21.15.46:0/2789658183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x561b3e7661f0 0x561b3e76ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.216+0000 7f4e1edcc740 1 --2- 172.21.15.46:0/2789658183 >> [v2:172.21.15.142:6838/3089509101,v1:172.21.15.142:6839/3089509101] conn(0x561b3e7e2c30 0x561b3e7df070 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.216+0000 7f4e1edcc740 1 --2- 172.21.15.46:0/2789658183 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x561b3e761080 0x561b3e76e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.216+0000 7f4e1edcc740 1 --2- 172.21.15.46:0/2789658183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b3e760690 0x561b3e771d50 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.216+0000 7f4e1edcc740 1 -- 172.21.15.46:0/2789658183 >> 172.21.15.46:0/2789658183 conn(0x561b3e75bd00 msgr2=0x561b3e75a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:41.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.216+0000 7f4dfd7fa640 0 client.15264 ms_handle_reset on v2:172.21.15.142:6838/3089509101 2024-04-26T21:34:41.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.216+0000 7f4e1edcc740 1 -- 172.21.15.46:0/2789658183 shutdown_connections 2024-04-26T21:34:41.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.216+0000 7f4e1edcc740 1 -- 172.21.15.46:0/2789658183 wait complete. 2024-04-26T21:34:41.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.216+0000 7f4e1edcc740 20 client.15264 trim_cache size 0 max 16384 2024-04-26T21:34:41.224 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-26T21:34:41.224 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36c7013a-0413-11ef-bc93-c7b262605968", 2024-04-26T21:34:41.224 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 1, 2024-04-26T21:34:41.224 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 24526, 2024-04-26T21:34:41.224 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-04-26T21:34:41.224 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-04-26T21:34:41.224 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-04-26T21:34:41.224 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 113.341842736, 2024-04-26T21:34:41.224 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 22, 2024-04-26T21:34:41.224 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 87, 2024-04-26T21:34:41.224 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 85, 2024-04-26T21:34:41.224 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 131.11808360200001 2024-04-26T21:34:41.224 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-26T21:34:41.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.217+0000 7f4e1bfda640 1 -- 172.21.15.46:0/1513749409 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4e000a3c40 msgr2=0x7f4e000a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.217+0000 7f4e1bfda640 1 --2- 172.21.15.46:0/1513749409 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4e000a3c40 0x7f4e000a6100 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c009270 tx=0x7f4e0c012c10 comp rx=0 tx=0).stop 2024-04-26T21:34:41.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.217+0000 7f4e1bfda640 1 -- 172.21.15.46:0/1513749409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e14105d40 msgr2=0x7f4e141991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.217+0000 7f4e1bfda640 1 --2- 172.21.15.46:0/1513749409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e14105d40 0x7f4e141991c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f4e04006010 tx=0x7f4e040042f0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.217+0000 7f4e1bfda640 1 -- 172.21.15.46:0/1513749409 shutdown_connections 2024-04-26T21:34:41.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.217+0000 7f4e1bfda640 1 --2- 172.21.15.46:0/1513749409 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4e000a3c40 0x7f4e000a6100 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.217+0000 7f4e1bfda640 1 --2- 172.21.15.46:0/1513749409 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e14108670 0x7f4e141a01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.217+0000 7f4e1bfda640 1 --2- 172.21.15.46:0/1513749409 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e14105d40 0x7f4e141991c0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.217+0000 7f4e1bfda640 1 --2- 172.21.15.46:0/1513749409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e14103410 0x7f4e14198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.217+0000 7f4e1bfda640 1 -- 172.21.15.46:0/1513749409 >> 172.21.15.46:0/1513749409 conn(0x7f4e140fce20 msgr2=0x7f4e14103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:41.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.217+0000 7f4e1bfda640 1 -- 172.21.15.46:0/1513749409 shutdown_connections 2024-04-26T21:34:41.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.217+0000 7f4e1bfda640 1 -- 172.21.15.46:0/1513749409 wait complete. 2024-04-26T21:34:41.229 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-26T21:34:41.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.330+0000 7fee7d59e640 1 -- 172.21.15.46:0/413585643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee78101870 msgr2=0x7fee78101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.330+0000 7fee7d59e640 1 --2- 172.21.15.46:0/413585643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee78101870 0x7fee78101c90 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fee68009900 tx=0x7fee6802a410 comp rx=0 tx=0).stop 2024-04-26T21:34:41.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.330+0000 7fee7d59e640 1 -- 172.21.15.46:0/413585643 shutdown_connections 2024-04-26T21:34:41.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.330+0000 7fee7d59e640 1 --2- 172.21.15.46:0/413585643 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee78103420 0x7fee78109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.330+0000 7fee7d59e640 1 --2- 172.21.15.46:0/413585643 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee78102a60 0x7fee78102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.330+0000 7fee7d59e640 1 --2- 172.21.15.46:0/413585643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee78101870 0x7fee78101c90 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.330+0000 7fee7d59e640 1 -- 172.21.15.46:0/413585643 >> 172.21.15.46:0/413585643 conn(0x7fee780fcde0 msgr2=0x7fee780ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:41.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.330+0000 7fee7d59e640 1 -- 172.21.15.46:0/413585643 shutdown_connections 2024-04-26T21:34:41.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.330+0000 7fee7d59e640 1 -- 172.21.15.46:0/413585643 wait complete. 2024-04-26T21:34:41.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.331+0000 7fee7d59e640 1 Processor -- start 2024-04-26T21:34:41.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.331+0000 7fee7d59e640 1 -- start start 2024-04-26T21:34:41.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.331+0000 7fee7d59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee78101870 0x7fee78105800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.331+0000 7fee7d59e640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee78102a60 0x7fee78103e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.331+0000 7fee7d59e640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee78103420 0x7fee78104380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.331+0000 7fee7d59e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fee7810d3f0 con 0x7fee78101870 2024-04-26T21:34:41.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.331+0000 7fee7d59e640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fee7810d270 con 0x7fee78102a60 2024-04-26T21:34:41.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.331+0000 7fee7d59e640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fee7810d570 con 0x7fee78103420 2024-04-26T21:34:41.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.331+0000 7fee76ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee78101870 0x7fee78105800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.332+0000 7fee76ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee78101870 0x7fee78105800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42116/0 (socket says 172.21.15.46:42116) 2024-04-26T21:34:41.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.332+0000 7fee76ffd640 1 -- 172.21.15.46:0/550246264 learned_addr learned my addr 172.21.15.46:0/550246264 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:41.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.332+0000 7fee777fe640 1 --2- 172.21.15.46:0/550246264 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee78103420 0x7fee78104380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.332+0000 7fee767fc640 1 --2- 172.21.15.46:0/550246264 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee78102a60 0x7fee78103e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.332+0000 7fee76ffd640 1 -- 172.21.15.46:0/550246264 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee78103420 msgr2=0x7fee78104380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.332+0000 7fee76ffd640 1 --2- 172.21.15.46:0/550246264 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee78103420 0x7fee78104380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.332+0000 7fee76ffd640 1 -- 172.21.15.46:0/550246264 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee78102a60 msgr2=0x7fee78103e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.332+0000 7fee76ffd640 1 --2- 172.21.15.46:0/550246264 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee78102a60 0x7fee78103e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.332+0000 7fee76ffd640 1 -- 172.21.15.46:0/550246264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee78104c10 con 0x7fee78101870 2024-04-26T21:34:41.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.332+0000 7fee76ffd640 1 --2- 172.21.15.46:0/550246264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee78101870 0x7fee78105800 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fee68006010 tx=0x7fee680042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:41.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.332+0000 7fee5ffff640 1 -- 172.21.15.46:0/550246264 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6803f030 con 0x7fee78101870 2024-04-26T21:34:41.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.332+0000 7fee7d59e640 1 -- 172.21.15.46:0/550246264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee781a2d60 con 0x7fee78101870 2024-04-26T21:34:41.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.332+0000 7fee5ffff640 1 -- 172.21.15.46:0/550246264 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee68003cb0 con 0x7fee78101870 2024-04-26T21:34:41.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.332+0000 7fee5ffff640 1 -- 172.21.15.46:0/550246264 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee68035070 con 0x7fee78101870 2024-04-26T21:34:41.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.332+0000 7fee7d59e640 1 -- 172.21.15.46:0/550246264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee781a31a0 con 0x7fee78101870 2024-04-26T21:34:41.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.335+0000 7fee5ffff640 1 -- 172.21.15.46:0/550246264 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fee68042030 con 0x7fee78101870 2024-04-26T21:34:41.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.335+0000 7fee8038a740 1 Processor -- start 2024-04-26T21:34:41.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.335+0000 7fee8038a740 1 -- start start 2024-04-26T21:34:41.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.335+0000 7fee8038a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570b5d09690 0x5570b5d09ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.335+0000 7fee8038a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d0a080 0x5570b5d0eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.335+0000 7fee8038a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5570b5d0f1f0 0x5570b5d11680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.335+0000 7fee8038a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5570b5c6e740 con 0x5570b5d0a080 2024-04-26T21:34:41.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.335+0000 7fee8038a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5570b5c0a3a0 con 0x5570b5d0f1f0 2024-04-26T21:34:41.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.335+0000 7fee8038a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5570b5c6a700 con 0x5570b5d09690 2024-04-26T21:34:41.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.335+0000 7fee76ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d0a080 0x5570b5d0eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.335+0000 7fee5ffff640 1 --2- 172.21.15.46:0/550246264 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fee600a3c40 0x7fee600a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.335+0000 7fee76ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d0a080 0x5570b5d0eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42124/0 (socket says 172.21.15.46:42124) 2024-04-26T21:34:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.335+0000 7fee76ffd640 1 -- 172.21.15.46:0/2944441207 learned_addr learned my addr 172.21.15.46:0/2944441207 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.335+0000 7fee767fc640 1 --2- 172.21.15.46:0/2944441207 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5570b5d0f1f0 0x5570b5d11680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.335+0000 7fee777fe640 1 --2- 172.21.15.46:0/2944441207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570b5d09690 0x5570b5d09ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.335+0000 7fee777fe640 1 --2- 172.21.15.46:0/550246264 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fee600a3c40 0x7fee600a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.335+0000 7fee5ffff640 1 -- 172.21.15.46:0/550246264 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7fee680f0ad0 con 0x7fee78101870 2024-04-26T21:34:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.336+0000 7fee76ffd640 1 -- 172.21.15.46:0/2944441207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570b5d09690 msgr2=0x5570b5d09ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.336+0000 7fee76ffd640 1 --2- 172.21.15.46:0/2944441207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570b5d09690 0x5570b5d09ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.336+0000 7fee76ffd640 1 -- 172.21.15.46:0/2944441207 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5570b5d0f1f0 msgr2=0x5570b5d11680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.336+0000 7fee76ffd640 1 --2- 172.21.15.46:0/2944441207 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5570b5d0f1f0 0x5570b5d11680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.336+0000 7fee76ffd640 1 -- 172.21.15.46:0/2944441207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570b5c26750 con 0x5570b5d0a080 2024-04-26T21:34:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.336+0000 7fee777fe640 1 --2- 172.21.15.46:0/2944441207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570b5d09690 0x5570b5d09ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:34:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.336+0000 7fee76ffd640 1 --2- 172.21.15.46:0/2944441207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d0a080 0x5570b5d0eb80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fee680ee420 tx=0x7fee680a78b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61f6102abb681883 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.336+0000 7fee777fe640 1 --2- 172.21.15.46:0/550246264 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fee600a3c40 0x7fee600a6100 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fee70009900 tx=0x7fee70009650 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.336+0000 7fee5d7fa640 1 -- 172.21.15.46:0/2944441207 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee68049070 con 0x5570b5d0a080 2024-04-26T21:34:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.336+0000 7fee5d7fa640 1 -- 172.21.15.46:0/2944441207 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee680bcd30 con 0x5570b5d0a080 2024-04-26T21:34:41.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.336+0000 7fee5d7fa640 1 -- 172.21.15.46:0/2944441207 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee680b93a0 con 0x5570b5d0a080 2024-04-26T21:34:41.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.337+0000 7fee8038a740 1 -- 172.21.15.46:0/2944441207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d0a080 msgr2=0x5570b5d0eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.337+0000 7fee8038a740 1 --2- 172.21.15.46:0/2944441207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d0a080 0x5570b5d0eb80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fee680ee420 tx=0x7fee680a78b0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.337+0000 7fee8038a740 1 -- 172.21.15.46:0/2944441207 shutdown_connections 2024-04-26T21:34:41.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.337+0000 7fee8038a740 1 --2- 172.21.15.46:0/2944441207 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5570b5d0f1f0 0x5570b5d11680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.337+0000 7fee8038a740 1 --2- 172.21.15.46:0/2944441207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d0a080 0x5570b5d0eb80 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.337+0000 7fee8038a740 1 --2- 172.21.15.46:0/2944441207 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570b5d09690 0x5570b5d09ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.337+0000 7fee8038a740 1 -- 172.21.15.46:0/2944441207 >> 172.21.15.46:0/2944441207 conn(0x5570b5d04d00 msgr2=0x5570b5d05100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:41.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.337+0000 7fee8038a740 1 -- 172.21.15.46:0/2944441207 shutdown_connections 2024-04-26T21:34:41.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.337+0000 7fee8038a740 1 -- 172.21.15.46:0/2944441207 wait complete. 2024-04-26T21:34:41.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.337+0000 7fee8038a740 1 -- start start 2024-04-26T21:34:41.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.337+0000 7fee8038a740 1 Processor -- start 2024-04-26T21:34:41.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.337+0000 7fee8038a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d09690 0x5570b5d136a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.337+0000 7fee8038a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570b5d0a080 0x5570b5d13be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.337+0000 7fee8038a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5570b5d0f1f0 0x5570b5d1ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.337+0000 7fee8038a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5570b5c6e740 con 0x5570b5d09690 2024-04-26T21:34:41.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.337+0000 7fee8038a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5570b5bf5fe0 con 0x5570b5d0f1f0 2024-04-26T21:34:41.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.337+0000 7fee8038a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5570b5d11f60 con 0x5570b5d0a080 2024-04-26T21:34:41.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.337+0000 7fee8038a740 10 client.0 _ll_register_callbacks cb 0x5570b5c0c150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:34:41.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.337+0000 7fee76ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d09690 0x5570b5d136a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.337+0000 7fee76ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d09690 0x5570b5d136a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42134/0 (socket says 172.21.15.46:42134) 2024-04-26T21:34:41.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.337+0000 7fee76ffd640 1 -- 172.21.15.46:0/3410008577 learned_addr learned my addr 172.21.15.46:0/3410008577 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:41.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.337+0000 7fee767fc640 1 --2- 172.21.15.46:0/3410008577 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570b5d0a080 0x5570b5d13be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.338+0000 7fee777fe640 1 --2- 172.21.15.46:0/3410008577 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5570b5d0f1f0 0x5570b5d1ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.338+0000 7fee76ffd640 1 -- 172.21.15.46:0/3410008577 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570b5d0a080 msgr2=0x5570b5d13be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:41.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.338+0000 7fee76ffd640 1 --2- 172.21.15.46:0/3410008577 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570b5d0a080 0x5570b5d13be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.338+0000 7fee76ffd640 1 -- 172.21.15.46:0/3410008577 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5570b5d0f1f0 msgr2=0x5570b5d1ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:41.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.338+0000 7fee76ffd640 1 --2- 172.21.15.46:0/3410008577 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5570b5d0f1f0 0x5570b5d1ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.338+0000 7fee76ffd640 1 -- 172.21.15.46:0/3410008577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570b59802d0 con 0x5570b5d09690 2024-04-26T21:34:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.338+0000 7fee76ffd640 1 --2- 172.21.15.46:0/3410008577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d09690 0x5570b5d136a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fee680c53c0 tx=0x7fee68009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b48407a53e16acb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.338+0000 7fee5aff5640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-26T21:34:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.338+0000 7fee5aff5640 1 -- 172.21.15.46:0/3410008577 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee680d6060 con 0x5570b5d09690 2024-04-26T21:34:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.338+0000 7fee5aff5640 1 -- 172.21.15.46:0/3410008577 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7fee680dd0b0 con 0x5570b5d09690 2024-04-26T21:34:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.338+0000 7fee8038a740 1 -- 172.21.15.46:0/3410008577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee680bcd30 con 0x5570b5d09690 2024-04-26T21:34:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.338+0000 7fee5aff5640 1 -- 172.21.15.46:0/3410008577 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fee68049050 con 0x5570b5d09690 2024-04-26T21:34:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.338+0000 7fee8038a740 10 client.15276 fetch_fsmap learned FSMap version 22 2024-04-26T21:34:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.338+0000 7fee8038a740 1 -- 172.21.15.46:0/3410008577 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5570b5a4a6b0 con 0x5570b5d09690 2024-04-26T21:34:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.338+0000 7fee5aff5640 1 -- 172.21.15.46:0/3410008577 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee680bcd30 con 0x5570b5d09690 2024-04-26T21:34:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.338+0000 7fee5aff5640 1 -- 172.21.15.46:0/3410008577 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fee680c5460 con 0x5570b5d09690 2024-04-26T21:34:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.338+0000 7fee8038a740 10 client.15276 fetch_fsmap finished waiting for FSMap version 22 2024-04-26T21:34:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.338+0000 7fee8038a740 10 client.15276 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T21:34:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.338+0000 7fee8038a740 20 client.15276 populate_metadata read hostname 'smithi046' 2024-04-26T21:34:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.338+0000 7fee8038a740 1 --2- 172.21.15.46:0/3410008577 >> [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] conn(0x5570b5d1fee0 0x5570b5d21350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.338+0000 7fee8038a740 4 client.15276 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:34:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.338+0000 7fee8038a740 1 -- 172.21.15.46:0/3410008577 --> [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570b5c7c350 con 0x5570b5d1fee0 2024-04-26T21:34:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.339+0000 7fee767fc640 1 --2- 172.21.15.46:0/3410008577 >> [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] conn(0x5570b5d1fee0 0x5570b5d21350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.339+0000 7fee767fc640 1 --2- 172.21.15.46:0/3410008577 >> [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] conn(0x5570b5d1fee0 0x5570b5d21350 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd1cab42d5f0a022 server_cookie=433720c4a3b551d8 in_seq=0 out_seq=0 2024-04-26T21:34:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.339+0000 7fee5aff5640 10 client.15276 ms_handle_connect on v2:172.21.15.50:6838/2453253368 2024-04-26T21:34:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.340+0000 7fee5aff5640 1 -- 172.21.15.46:0/3410008577 <== mds.0 v2:172.21.15.50:6838/2453253368 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5570b5c7c350 con 0x5570b5d1fee0 2024-04-26T21:34:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.340+0000 7fee5aff5640 10 client.15276 handle_command_reply: tid=0 2024-04-26T21:34:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.340+0000 7fee8038a740 1 client.15276 shutdown 2024-04-26T21:34:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.340+0000 7fee8038a740 1 -- 172.21.15.46:0/3410008577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d09690 msgr2=0x5570b5d136a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.340+0000 7fee8038a740 1 --2- 172.21.15.46:0/3410008577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d09690 0x5570b5d136a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fee680c53c0 tx=0x7fee68009260 comp rx=0 tx=0).stop 2024-04-26T21:34:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.340+0000 7fee8038a740 1 -- 172.21.15.46:0/3410008577 shutdown_connections 2024-04-26T21:34:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.340+0000 7fee8038a740 1 --2- 172.21.15.46:0/3410008577 >> [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] conn(0x5570b5d1fee0 0x5570b5d21350 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.340+0000 7fee8038a740 1 --2- 172.21.15.46:0/3410008577 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5570b5d0f1f0 0x5570b5d1ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.340+0000 7fee5aff5640 0 client.15276 ms_handle_reset on v2:172.21.15.50:6838/2453253368 2024-04-26T21:34:41.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.340+0000 7fee8038a740 1 --2- 172.21.15.46:0/3410008577 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570b5d0a080 0x5570b5d13be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.340+0000 7fee8038a740 1 --2- 172.21.15.46:0/3410008577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d09690 0x5570b5d136a0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.340+0000 7fee8038a740 1 -- 172.21.15.46:0/3410008577 >> 172.21.15.46:0/3410008577 conn(0x5570b5d04d00 msgr2=0x5570b5d03f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:41.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.340+0000 7fee8038a740 1 -- 172.21.15.46:0/3410008577 shutdown_connections 2024-04-26T21:34:41.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.340+0000 7fee8038a740 1 -- 172.21.15.46:0/3410008577 wait complete. 2024-04-26T21:34:41.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.340+0000 7fee8038a740 20 client.15276 trim_cache size 0 max 16384 2024-04-26T21:34:41.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.358+0000 7fee8038a740 1 Processor -- start 2024-04-26T21:34:41.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.358+0000 7fee8038a740 1 -- start start 2024-04-26T21:34:41.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.358+0000 7fee8038a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570b5d04d00 0x5570b5d132b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.358+0000 7fee8038a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d09690 0x5570b5d1abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.358+0000 7fee8038a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5570b5d0a080 0x5570b5d137f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.358+0000 7fee8038a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5570b5d1f690 con 0x5570b5d09690 2024-04-26T21:34:41.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.358+0000 7fee8038a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5570b5d1fd60 con 0x5570b5d0a080 2024-04-26T21:34:41.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.358+0000 7fee8038a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5570b5d20430 con 0x5570b5d04d00 2024-04-26T21:34:41.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.358+0000 7fee767fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d09690 0x5570b5d1abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.358+0000 7fee767fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d09690 0x5570b5d1abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42144/0 (socket says 172.21.15.46:42144) 2024-04-26T21:34:41.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.358+0000 7fee767fc640 1 -- 172.21.15.46:0/3150507115 learned_addr learned my addr 172.21.15.46:0/3150507115 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:41.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.358+0000 7fee777fe640 1 --2- 172.21.15.46:0/3150507115 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5570b5d0a080 0x5570b5d137f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.358+0000 7fee767fc640 1 -- 172.21.15.46:0/3150507115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570b5d04d00 msgr2=0x5570b5d132b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:41.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.358+0000 7fee767fc640 1 --2- 172.21.15.46:0/3150507115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570b5d04d00 0x5570b5d132b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.358+0000 7fee767fc640 1 -- 172.21.15.46:0/3150507115 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5570b5d0a080 msgr2=0x5570b5d137f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:41.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.359+0000 7fee767fc640 1 --2- 172.21.15.46:0/3150507115 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5570b5d0a080 0x5570b5d137f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.359+0000 7fee767fc640 1 -- 172.21.15.46:0/3150507115 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570b5d0a460 con 0x5570b5d09690 2024-04-26T21:34:41.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.359+0000 7fee777fe640 1 --2- 172.21.15.46:0/3150507115 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5570b5d0a080 0x5570b5d137f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:34:41.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.359+0000 7fee767fc640 1 --2- 172.21.15.46:0/3150507115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d09690 0x5570b5d1abb0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fee6c005d00 tx=0x7fee6c00b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51606111d7899f46 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:41.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.359+0000 7fee5cff9640 1 -- 172.21.15.46:0/3150507115 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6c004220 con 0x5570b5d09690 2024-04-26T21:34:41.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.359+0000 7fee5cff9640 1 -- 172.21.15.46:0/3150507115 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee6c023070 con 0x5570b5d09690 2024-04-26T21:34:41.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.359+0000 7fee5cff9640 1 -- 172.21.15.46:0/3150507115 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6c01f720 con 0x5570b5d09690 2024-04-26T21:34:41.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.359+0000 7fee8038a740 1 -- 172.21.15.46:0/3150507115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d09690 msgr2=0x5570b5d1abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.359+0000 7fee8038a740 1 --2- 172.21.15.46:0/3150507115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d09690 0x5570b5d1abb0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fee6c005d00 tx=0x7fee6c00b410 comp rx=0 tx=0).stop 2024-04-26T21:34:41.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.359+0000 7fee8038a740 1 -- 172.21.15.46:0/3150507115 shutdown_connections 2024-04-26T21:34:41.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.359+0000 7fee8038a740 1 --2- 172.21.15.46:0/3150507115 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5570b5d0a080 0x5570b5d137f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.359+0000 7fee8038a740 1 --2- 172.21.15.46:0/3150507115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d09690 0x5570b5d1abb0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.359+0000 7fee8038a740 1 --2- 172.21.15.46:0/3150507115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570b5d04d00 0x5570b5d132b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.359+0000 7fee8038a740 1 -- 172.21.15.46:0/3150507115 >> 172.21.15.46:0/3150507115 conn(0x5570b5d0ae60 msgr2=0x5570b5d121e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:41.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.359+0000 7fee8038a740 1 -- 172.21.15.46:0/3150507115 shutdown_connections 2024-04-26T21:34:41.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.359+0000 7fee8038a740 1 -- 172.21.15.46:0/3150507115 wait complete. 2024-04-26T21:34:41.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.360+0000 7fee8038a740 1 -- start start 2024-04-26T21:34:41.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.360+0000 7fee8038a740 1 Processor -- start 2024-04-26T21:34:41.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.360+0000 7fee8038a740 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5570b5d04d00 0x5570b5d17730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.360+0000 7fee8038a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d09690 0x5570b5d17c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.360+0000 7fee8038a740 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570b5d0a080 0x5570b5d85e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.360+0000 7fee8038a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x5570b5d211d0 con 0x5570b5d09690 2024-04-26T21:34:41.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.360+0000 7fee8038a740 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x5570b5d20b00 con 0x5570b5d04d00 2024-04-26T21:34:41.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.360+0000 7fee8038a740 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x5570b5d218a0 con 0x5570b5d0a080 2024-04-26T21:34:41.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.360+0000 7fee8038a740 10 client.0 _ll_register_callbacks cb 0x7fee6c00b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:34:41.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.360+0000 7fee767fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d09690 0x5570b5d17c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.360+0000 7fee767fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d09690 0x5570b5d17c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42154/0 (socket says 172.21.15.46:42154) 2024-04-26T21:34:41.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.360+0000 7fee767fc640 1 -- 172.21.15.46:0/1374509222 learned_addr learned my addr 172.21.15.46:0/1374509222 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:41.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.360+0000 7fee767fc640 1 -- 172.21.15.46:0/1374509222 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570b5d0a080 msgr2=0x5570b5d85e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:41.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.360+0000 7fee767fc640 1 --2- 172.21.15.46:0/1374509222 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570b5d0a080 0x5570b5d85e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.360+0000 7fee767fc640 1 -- 172.21.15.46:0/1374509222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5570b5d04d00 msgr2=0x5570b5d17730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:41.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.360+0000 7fee767fc640 1 --2- 172.21.15.46:0/1374509222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5570b5d04d00 0x5570b5d17730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.360+0000 7fee767fc640 1 -- 172.21.15.46:0/1374509222 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570b5d1ad40 con 0x5570b5d09690 2024-04-26T21:34:41.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.360+0000 7fee767fc640 1 --2- 172.21.15.46:0/1374509222 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d09690 0x5570b5d17c70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fee6c022040 tx=0x7fee6c01f250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e405076c7c4da24 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:41.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.360+0000 7fee5aff5640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-04-26T21:34:41.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.361+0000 7fee5aff5640 1 -- 172.21.15.46:0/1374509222 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6c01f4a0 con 0x5570b5d09690 2024-04-26T21:34:41.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.361+0000 7fee5aff5640 1 -- 172.21.15.46:0/1374509222 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7fee6c01f640 con 0x5570b5d09690 2024-04-26T21:34:41.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.361+0000 7fee8038a740 1 -- 172.21.15.46:0/1374509222 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee6c023070 con 0x5570b5d09690 2024-04-26T21:34:41.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.361+0000 7fee5aff5640 1 -- 172.21.15.46:0/1374509222 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fee6c037050 con 0x5570b5d09690 2024-04-26T21:34:41.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.361+0000 7fee8038a740 10 client.15282 fetch_fsmap learned FSMap version 22 2024-04-26T21:34:41.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.361+0000 7fee8038a740 1 -- 172.21.15.46:0/1374509222 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) -- 0x5570b5d87200 con 0x5570b5d09690 2024-04-26T21:34:41.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.361+0000 7fee5aff5640 1 -- 172.21.15.46:0/1374509222 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee6c023070 con 0x5570b5d09690 2024-04-26T21:34:41.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.361+0000 7fee5aff5640 1 -- 172.21.15.46:0/1374509222 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 22) ==== 8338+0+0 (secure 0 0 0) 0x7fee6c003ae0 con 0x5570b5d09690 2024-04-26T21:34:41.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.361+0000 7fee8038a740 10 client.15282 fetch_fsmap finished waiting for FSMap version 22 2024-04-26T21:34:41.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.361+0000 7fee8038a740 10 client.15282 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T21:34:41.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.361+0000 7fee8038a740 20 client.15282 populate_metadata read hostname 'smithi046' 2024-04-26T21:34:41.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.361+0000 7fee8038a740 1 --2- 172.21.15.46:0/1374509222 >> [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] conn(0x5570b5d0f1f0 0x5570b5d8a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:41.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.361+0000 7fee8038a740 4 client.15282 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T21:34:41.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.361+0000 7fee8038a740 1 -- 172.21.15.46:0/1374509222 --> [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] -- command(tid 0: {"prefix": "status"}) -- 0x5570b5c761c0 con 0x5570b5d0f1f0 2024-04-26T21:34:41.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.361+0000 7fee76ffd640 1 --2- 172.21.15.46:0/1374509222 >> [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] conn(0x5570b5d0f1f0 0x5570b5d8a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:41.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.362+0000 7fee76ffd640 1 --2- 172.21.15.46:0/1374509222 >> [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] conn(0x5570b5d0f1f0 0x5570b5d8a7c0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cba0eeedb6cbfa83 server_cookie=946f7267efc4addd in_seq=0 out_seq=0 2024-04-26T21:34:41.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.362+0000 7fee5aff5640 10 client.15282 ms_handle_connect on v2:172.21.15.50:6838/2453253368 2024-04-26T21:34:41.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.362+0000 7fee5aff5640 1 -- 172.21.15.46:0/1374509222 <== mds.0 v2:172.21.15.50:6838/2453253368 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5570b5c761c0 con 0x5570b5d0f1f0 2024-04-26T21:34:41.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.362+0000 7fee5aff5640 10 client.15282 handle_command_reply: tid=0 2024-04-26T21:34:41.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.362+0000 7fee8038a740 1 client.15282 shutdown 2024-04-26T21:34:41.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.362+0000 7fee8038a740 1 -- 172.21.15.46:0/1374509222 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d09690 msgr2=0x5570b5d17c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.362+0000 7fee8038a740 1 --2- 172.21.15.46:0/1374509222 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d09690 0x5570b5d17c70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fee6c022040 tx=0x7fee6c01f250 comp rx=0 tx=0).stop 2024-04-26T21:34:41.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.362+0000 7fee8038a740 1 -- 172.21.15.46:0/1374509222 shutdown_connections 2024-04-26T21:34:41.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.362+0000 7fee8038a740 1 --2- 172.21.15.46:0/1374509222 >> [v2:172.21.15.50:6838/2453253368,v1:172.21.15.50:6839/2453253368] conn(0x5570b5d0f1f0 0x5570b5d8a7c0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.362+0000 7fee8038a740 1 --2- 172.21.15.46:0/1374509222 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x5570b5d0a080 0x5570b5d85e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.362+0000 7fee5aff5640 0 client.15282 ms_handle_reset on v2:172.21.15.50:6838/2453253368 2024-04-26T21:34:41.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.362+0000 7fee8038a740 1 --2- 172.21.15.46:0/1374509222 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570b5d09690 0x5570b5d17c70 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.362+0000 7fee8038a740 1 --2- 172.21.15.46:0/1374509222 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x5570b5d04d00 0x5570b5d17730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.362+0000 7fee8038a740 1 -- 172.21.15.46:0/1374509222 >> 172.21.15.46:0/1374509222 conn(0x5570b5d0ae60 msgr2=0x5570b5d13290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:41.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.362+0000 7fee8038a740 1 -- 172.21.15.46:0/1374509222 shutdown_connections 2024-04-26T21:34:41.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.362+0000 7fee8038a740 1 -- 172.21.15.46:0/1374509222 wait complete. 2024-04-26T21:34:41.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.362+0000 7fee8038a740 20 client.15282 trim_cache size 0 max 16384 2024-04-26T21:34:41.370 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-04-26T21:34:41.370 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "36c7013a-0413-11ef-bc93-c7b262605968", 2024-04-26T21:34:41.370 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-04-26T21:34:41.371 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 24539, 2024-04-26T21:34:41.371 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-04-26T21:34:41.371 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-04-26T21:34:41.371 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-04-26T21:34:41.371 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 116.40362578600001, 2024-04-26T21:34:41.371 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 22, 2024-04-26T21:34:41.371 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 87, 2024-04-26T21:34:41.371 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 85, 2024-04-26T21:34:41.371 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 122.289822808 2024-04-26T21:34:41.371 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-04-26T21:34:41.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.363+0000 7fee7d59e640 1 -- 172.21.15.46:0/550246264 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fee600a3c40 msgr2=0x7fee600a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.363+0000 7fee7d59e640 1 --2- 172.21.15.46:0/550246264 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fee600a3c40 0x7fee600a6100 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fee70009900 tx=0x7fee70009650 comp rx=0 tx=0).stop 2024-04-26T21:34:41.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.363+0000 7fee7d59e640 1 -- 172.21.15.46:0/550246264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee78101870 msgr2=0x7fee78105800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:41.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.363+0000 7fee7d59e640 1 --2- 172.21.15.46:0/550246264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee78101870 0x7fee78105800 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fee68006010 tx=0x7fee680042f0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.363+0000 7fee7d59e640 1 -- 172.21.15.46:0/550246264 shutdown_connections 2024-04-26T21:34:41.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.363+0000 7fee7d59e640 1 --2- 172.21.15.46:0/550246264 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fee600a3c40 0x7fee600a6100 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.363+0000 7fee7d59e640 1 --2- 172.21.15.46:0/550246264 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fee78103420 0x7fee78104380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.363+0000 7fee7d59e640 1 --2- 172.21.15.46:0/550246264 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee78102a60 0x7fee78103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.363+0000 7fee7d59e640 1 --2- 172.21.15.46:0/550246264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee78101870 0x7fee78105800 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:41.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.363+0000 7fee7d59e640 1 -- 172.21.15.46:0/550246264 >> 172.21.15.46:0/550246264 conn(0x7fee780fcde0 msgr2=0x7fee78069b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:41.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.363+0000 7fee7d59e640 1 -- 172.21.15.46:0/550246264 shutdown_connections 2024-04-26T21:34:41.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:41.363+0000 7fee7d59e640 1 -- 172.21.15.46:0/550246264 wait complete. 2024-04-26T21:34:41.375 INFO:teuthology.run_tasks:Running task sequential... 2024-04-26T21:34:41.384 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-26T21:34:41.384 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-26T21:34:41.384 INFO:teuthology.task.print:Enabling mgr modules 2024-04-26T21:34:41.385 INFO:teuthology.run_tasks:Running task kclient... 2024-04-26T21:34:41.406 INFO:tasks.kclient:Mounting kernel clients... 2024-04-26T21:34:41.407 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-26T21:34:41.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:34:41.407 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-04-26T21:34:41.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:34:41.423 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link delete ceph-brx 2024-04-26T21:34:41.488 INFO:teuthology.orchestra.run.smithi046.stderr:Cannot find device "ceph-brx" 2024-04-26T21:34:41.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T21:34:41.490 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-04-26T21:34:41.490 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-26T21:34:41.490 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-26T21:34:41.491 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 36c7013a-0413-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-26T21:34:41.775 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:34:41.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:41 smithi046 ceph-mon[29201]: pgmap v352: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:34:41.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:41 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1367623893' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:34:42.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:41 smithi142 ceph-mon[32944]: pgmap v352: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:34:42.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:41 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/1367623893' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:34:42.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:41 smithi050 ceph-mon[32753]: pgmap v352: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:34:42.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:41 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/1367623893' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:34:43.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.585+0000 7f8cc4b21700 1 -- 172.21.15.46:0/2059852903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cc0103470 msgr2=0x7f8cc01038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:43.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.585+0000 7f8cc4b21700 1 --2- 172.21.15.46:0/2059852903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cc0103470 0x7f8cc01038f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4009990 tx=0x7f8cb4009ca0 comp rx=0 tx=0).stop 2024-04-26T21:34:43.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.585+0000 7f8cc4b21700 1 -- 172.21.15.46:0/2059852903 shutdown_connections 2024-04-26T21:34:43.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.585+0000 7f8cc4b21700 1 --2- 172.21.15.46:0/2059852903 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8cc0103e30 0x7f8cc0110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:43.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.585+0000 7f8cc4b21700 1 --2- 172.21.15.46:0/2059852903 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8cc01094c0 0x7f8cc01098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:43.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.585+0000 7f8cc4b21700 1 --2- 172.21.15.46:0/2059852903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cc0103470 0x7f8cc01038f0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:43.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.585+0000 7f8cc4b21700 1 -- 172.21.15.46:0/2059852903 >> 172.21.15.46:0/2059852903 conn(0x7f8cc007a580 msgr2=0x7f8cc007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:43.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.585+0000 7f8cc4b21700 1 -- 172.21.15.46:0/2059852903 shutdown_connections 2024-04-26T21:34:43.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.585+0000 7f8cc4b21700 1 -- 172.21.15.46:0/2059852903 wait complete. 2024-04-26T21:34:43.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.586+0000 7f8cc4b21700 1 Processor -- start 2024-04-26T21:34:43.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.586+0000 7f8cc4b21700 1 -- start start 2024-04-26T21:34:43.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.586+0000 7f8cc4b21700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8cc0103470 0x7f8cc010bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:43.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.586+0000 7f8cc4b21700 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8cc0103e30 0x7f8cc010c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:43.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.586+0000 7f8cc4b21700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cc01094c0 0x7f8cc010c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:43.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.586+0000 7f8cc4b21700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc01135c0 con 0x7f8cc01094c0 2024-04-26T21:34:43.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.586+0000 7f8cc4b21700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc0113440 con 0x7f8cc0103470 2024-04-26T21:34:43.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.586+0000 7f8cc4b21700 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc0113740 con 0x7f8cc0103e30 2024-04-26T21:34:43.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.586+0000 7f8cbed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cc01094c0 0x7f8cc010c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:43.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.586+0000 7f8cbed9d700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cc01094c0 0x7f8cc010c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42172/0 (socket says 172.21.15.46:42172) 2024-04-26T21:34:43.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.586+0000 7f8cbed9d700 1 -- 172.21.15.46:0/2006778121 learned_addr learned my addr 172.21.15.46:0/2006778121 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:43.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.586+0000 7f8cbe59c700 1 --2- 172.21.15.46:0/2006778121 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8cc0103470 0x7f8cc010bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:43.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.586+0000 7f8cbdd9b700 1 --2- 172.21.15.46:0/2006778121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8cc0103e30 0x7f8cc010c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:43.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.586+0000 7f8cbed9d700 1 -- 172.21.15.46:0/2006778121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8cc0103e30 msgr2=0x7f8cc010c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:43.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.586+0000 7f8cbed9d700 1 --2- 172.21.15.46:0/2006778121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8cc0103e30 0x7f8cc010c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:43.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.586+0000 7f8cbed9d700 1 -- 172.21.15.46:0/2006778121 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8cc0103470 msgr2=0x7f8cc010bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:43.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.586+0000 7f8cbed9d700 1 --2- 172.21.15.46:0/2006778121 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8cc0103470 0x7f8cc010bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:43.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.586+0000 7f8cbed9d700 1 -- 172.21.15.46:0/2006778121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cc01a1c30 con 0x7f8cc01094c0 2024-04-26T21:34:43.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.587+0000 7f8cbed9d700 1 --2- 172.21.15.46:0/2006778121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cc01094c0 0x7f8cc010c6d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8cb8007ee0 tx=0x7f8cb800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:43.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.587+0000 7f8caf7fe700 1 -- 172.21.15.46:0/2006778121 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb800ddd0 con 0x7f8cc01094c0 2024-04-26T21:34:43.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.587+0000 7f8cc4b21700 1 -- 172.21.15.46:0/2006778121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cc01a1f20 con 0x7f8cc01094c0 2024-04-26T21:34:43.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.587+0000 7f8caf7fe700 1 -- 172.21.15.46:0/2006778121 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8cb8004020 con 0x7f8cc01094c0 2024-04-26T21:34:43.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.587+0000 7f8caf7fe700 1 -- 172.21.15.46:0/2006778121 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8015660 con 0x7f8cc01094c0 2024-04-26T21:34:43.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.587+0000 7f8cc4b21700 1 -- 172.21.15.46:0/2006778121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cc01a2460 con 0x7f8cc01094c0 2024-04-26T21:34:43.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.588+0000 7f8cc4b21700 1 -- 172.21.15.46:0/2006778121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cc0104b60 con 0x7f8cc01094c0 2024-04-26T21:34:43.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.589+0000 7f8caf7fe700 1 -- 172.21.15.46:0/2006778121 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8cb80044f0 con 0x7f8cc01094c0 2024-04-26T21:34:43.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.590+0000 7f8caf7fe700 1 --2- 172.21.15.46:0/2006778121 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8ca80a3cd0 0x7f8ca80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:43.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.590+0000 7f8cbe59c700 1 --2- 172.21.15.46:0/2006778121 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8ca80a3cd0 0x7f8ca80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:43.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.590+0000 7f8caf7fe700 1 -- 172.21.15.46:0/2006778121 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f8cb80d0890 con 0x7f8cc01094c0 2024-04-26T21:34:43.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.590+0000 7f8cbe59c700 1 --2- 172.21.15.46:0/2006778121 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8ca80a3cd0 0x7f8ca80a6190 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8cb0005fd0 tx=0x7f8cb0005db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:43.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.593+0000 7f8caf7fe700 1 -- 172.21.15.46:0/2006778121 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8cb8010070 con 0x7f8cc01094c0 2024-04-26T21:34:43.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:43 smithi046 ceph-mon[29201]: pgmap v353: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:43.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.850+0000 7f8cc4b21700 1 -- 172.21.15.46:0/2006778121 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f8cc010d180 con 0x7f8cc01094c0 2024-04-26T21:34:43.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.850+0000 7f8caf7fe700 1 -- 172.21.15.46:0/2006778121 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v22) ==== 53+0+73 (secure 0 0 0) 0x7f8cb809d0e0 con 0x7f8cc01094c0 2024-04-26T21:34:43.852 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-26T21:34:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.853+0000 7f8cc4b21700 1 -- 172.21.15.46:0/2006778121 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8ca80a3cd0 msgr2=0x7f8ca80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.853+0000 7f8cc4b21700 1 --2- 172.21.15.46:0/2006778121 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8ca80a3cd0 0x7f8ca80a6190 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8cb0005fd0 tx=0x7f8cb0005db0 comp rx=0 tx=0).stop 2024-04-26T21:34:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.853+0000 7f8cc4b21700 1 -- 172.21.15.46:0/2006778121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cc01094c0 msgr2=0x7f8cc010c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.853+0000 7f8cc4b21700 1 --2- 172.21.15.46:0/2006778121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cc01094c0 0x7f8cc010c6d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8cb8007ee0 tx=0x7f8cb800d3b0 comp rx=0 tx=0).stop 2024-04-26T21:34:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.853+0000 7f8cc4b21700 1 -- 172.21.15.46:0/2006778121 shutdown_connections 2024-04-26T21:34:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.853+0000 7f8cc4b21700 1 --2- 172.21.15.46:0/2006778121 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f8ca80a3cd0 0x7f8ca80a6190 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.853+0000 7f8cc4b21700 1 --2- 172.21.15.46:0/2006778121 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cc01094c0 0x7f8cc010c6d0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.853+0000 7f8cc4b21700 1 --2- 172.21.15.46:0/2006778121 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8cc0103470 0x7f8cc010bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.853+0000 7f8cc4b21700 1 --2- 172.21.15.46:0/2006778121 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8cc0103e30 0x7f8cc010c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.853+0000 7f8cc4b21700 1 -- 172.21.15.46:0/2006778121 >> 172.21.15.46:0/2006778121 conn(0x7f8cc007a580 msgr2=0x7f8cc0102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.853+0000 7f8cc4b21700 1 -- 172.21.15.46:0/2006778121 shutdown_connections 2024-04-26T21:34:43.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:43.853+0000 7f8cc4b21700 1 -- 172.21.15.46:0/2006778121 wait complete. 2024-04-26T21:34:44.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:43 smithi142 ceph-mon[32944]: pgmap v353: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:44.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:43 smithi050 ceph-mon[32753]: pgmap v353: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:44.501 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-26T21:34:44.501 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T21:34:44.502 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-04-26T21:34:44.502 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-26T21:34:44.502 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-26T21:34:44.502 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-26T21:34:44.502 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-26T21:34:44.502 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T21:34:44.502 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-26T21:34:44.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:34:44.502 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-04-26T21:34:44.519 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T21:34:44.519 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T21:34:44.519 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T21:34:44.519 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-26T21:34:44.519 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-04-26T21:34:44.519 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-26T21:34:44.519 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T21:34:44.520 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-04-26T21:34:44.520 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-04-26T21:34:44.520 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T21:34:44.520 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-04-26T21:34:44.520 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-04-26T21:34:44.520 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T21:34:44.520 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-04-26T21:34:44.520 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-04-26T21:34:44.520 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-26T21:34:44.520 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42110sec preferred_lft 42110sec 2024-04-26T21:34:44.520 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-04-26T21:34:44.520 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-04-26T21:34:44.520 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T21:34:44.520 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-04-26T21:34:44.520 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-04-26T21:34:44.521 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-26T21:34:44.521 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T21:34:44.521 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-26T21:34:44.521 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add name ceph-brx type bridge 2024-04-26T21:34:44.521 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip addr flush dev ceph-brx 2024-04-26T21:34:44.521 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set ceph-brx up 2024-04-26T21:34:44.521 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-26T21:34:44.521 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-26T21:34:44.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T21:34:44.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T21:34:44.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:34:44.722 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T21:34:44.736 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:44 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2006778121' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T21:34:44.746 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-04-26T21:34:44.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:34:44.747 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-04-26T21:34:44.815 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-04-26T21:34:44.816 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T21:34:44.816 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-26T21:34:44.816 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-26T21:34:44.816 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T21:34:44.816 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T21:34:44.816 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-26T21:34:44.816 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-26T21:34:44.816 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-26T21:34:44.817 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-26T21:34:44.817 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-26T21:34:44.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T21:34:45.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:44 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/2006778121' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T21:34:45.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T21:34:45.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:44 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/2006778121' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T21:34:45.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:34:45.123 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-04-26T21:34:45.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:34:45.138 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list-id 2024-04-26T21:34:45.194 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T21:34:45.194 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-26T21:34:45.195 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-26T21:34:45.195 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-26T21:34:45.195 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-26T21:34:45.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T21:34:45.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T21:34:45.313 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-26T21:34:45.313 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T21:34:45.314 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-26T21:34:45.314 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-26T21:34:45.314 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-26T21:34:45.314 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-26T21:34:45.314 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-26T21:34:45.314 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-26T21:34:45.314 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-26T21:34:45.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T21:34:45.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T21:34:45.440 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T21:34:45.440 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-04-26T21:34:45.440 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-04-26T21:34:45.440 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set dev brx.0 master ceph-brx 2024-04-26T21:34:45.440 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-04-26T21:34:45.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T21:34:45.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T21:34:45.519 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T21:34:45.519 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-26T21:34:45.534 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-26T21:34:45.535 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T21:34:45.535 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-26T21:34:45.590 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-26T21:34:45.590 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-26T21:34:45.590 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/1453fde9-3580-415d-8a6d-8b422bc1e71d 2024-04-26T21:34:45.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:34:45.590 DEBUG:teuthology.orchestra.run.smithi046:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/1453fde9-3580-415d-8a6d-8b422bc1e71d /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-04-26T21:34:45.677 INFO:teuthology.orchestra.run.smithi046.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-04-26T21:34:45.677 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync". 2024-04-26T21:34:45.788 INFO:teuthology.orchestra.run.smithi046.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-04-26T21:34:45.788 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync". 2024-04-26T21:34:45.788 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: resolved to: "172.21.15.46:6789,172.21.15.50:6789,172.21.15.142:6789" 2024-04-26T21:34:45.788 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: trying mount with new device syntax: 0@36c7013a-0413-11ef-bc93-c7b262605968.cephfs=/volumes/qa/sv_1/1453fde9-3580-415d-8a6d-8b422bc1e71d 2024-04-26T21:34:45.788 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync,name=0,key=0,mon_addr=172.21.15.46:6789/172.21.15.50:6789/172.21.15.142:6789" will pass to kernel 2024-04-26T21:34:45.789 INFO:teuthology.orchestra.run.smithi046.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-26T21:34:45.789 INFO:teuthology.orchestra.run.smithi046.stdout: You just mounted a file system that supports labels which does not 2024-04-26T21:34:45.790 INFO:teuthology.orchestra.run.smithi046.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-26T21:34:45.790 INFO:teuthology.orchestra.run.smithi046.stdout: applications will generate AVC messages and not be allowed access to 2024-04-26T21:34:45.790 INFO:teuthology.orchestra.run.smithi046.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-26T21:34:45.792 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-26T21:34:45.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:34:45.792 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-26T21:34:45.828 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T21:34:45.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T21:34:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T21:34:45.927 INFO:teuthology.orchestra.run.smithi046.stdout:client24949 2024-04-26T21:34:45.927 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-26T21:34:45.927 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T21:34:45.943 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T21:34:46.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T21:34:46.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T21:34:46.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:45 smithi142 ceph-mon[32944]: pgmap v354: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:46.041 INFO:teuthology.orchestra.run.smithi046.stdout:client24949 2024-04-26T21:34:46.041 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T21:34:46.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:45 smithi046 ceph-mon[29201]: pgmap v354: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:46.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T21:34:46.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T21:34:46.098 INFO:teuthology.orchestra.run.smithi046.stdout:client24949 2024-04-26T21:34:46.098 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/36c7013a-0413-11ef-bc93-c7b262605968.client24949/status') 2024-04-26T21:34:46.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:45 smithi050 ceph-mon[32753]: pgmap v354: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:46.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T21:34:46.197 INFO:teuthology.orchestra.run.smithi046.stdout:instance: client.24949 (3)172.21.15.46:0/1927220926 2024-04-26T21:34:46.198 INFO:teuthology.orchestra.run.smithi046.stdout:blocklisted: false 2024-04-26T21:34:46.198 INFO:teuthology.orchestra.run.smithi046.stderr:0+1 records in 2024-04-26T21:34:46.198 INFO:teuthology.orchestra.run.smithi046.stderr:0+1 records out 2024-04-26T21:34:46.198 INFO:teuthology.orchestra.run.smithi046.stderr:71 bytes copied, 2.7839e-05 s, 2.6 MB/s 2024-04-26T21:34:46.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T21:34:46.205 INFO:teuthology.run_tasks:Running task exec... 2024-04-26T21:34:46.215 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T21:34:46.215 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi046.front.sepia.ceph.com 2024-04-26T21:34:46.216 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-04-26T21:34:46.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.317+0000 7f9f87f66640 1 -- 172.21.15.46:0/2809466641 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f80103bb0 msgr2=0x7f9f80104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:46.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.317+0000 7f9f87f66640 1 --2- 172.21.15.46:0/2809466641 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f80103bb0 0x7f9f80104030 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9f70009900 tx=0x7f9f7002a410 comp rx=0 tx=0).stop 2024-04-26T21:34:46.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.318+0000 7f9f87f66640 1 -- 172.21.15.46:0/2809466641 shutdown_connections 2024-04-26T21:34:46.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.318+0000 7f9f87f66640 1 --2- 172.21.15.46:0/2809466641 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f80104570 0x7f9f8010ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.318+0000 7f9f87f66640 1 --2- 172.21.15.46:0/2809466641 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f80103bb0 0x7f9f80104030 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.318+0000 7f9f87f66640 1 --2- 172.21.15.46:0/2809466641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f801029c0 0x7f9f80102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.318+0000 7f9f87f66640 1 -- 172.21.15.46:0/2809466641 >> 172.21.15.46:0/2809466641 conn(0x7f9f800fdf50 msgr2=0x7f9f80100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:46.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.318+0000 7f9f87f66640 1 -- 172.21.15.46:0/2809466641 shutdown_connections 2024-04-26T21:34:46.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.318+0000 7f9f87f66640 1 -- 172.21.15.46:0/2809466641 wait complete. 2024-04-26T21:34:46.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.318+0000 7f9f87f66640 1 Processor -- start 2024-04-26T21:34:46.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.318+0000 7f9f87f66640 1 -- start start 2024-04-26T21:34:46.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.318+0000 7f9f87f66640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f801029c0 0x7f9f80198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:46.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.318+0000 7f9f87f66640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f80103bb0 0x7f9f80198fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:46.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.318+0000 7f9f87f66640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f80104570 0x7f9f801a0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:46.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.318+0000 7f9f87f66640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8010e520 con 0x7f9f801029c0 2024-04-26T21:34:46.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.318+0000 7f9f87f66640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8010e3a0 con 0x7f9f80103bb0 2024-04-26T21:34:46.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.318+0000 7f9f87f66640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8010e6a0 con 0x7f9f80104570 2024-04-26T21:34:46.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.319+0000 7f9f85d02640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f801029c0 0x7f9f80198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:46.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.319+0000 7f9f85d02640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f801029c0 0x7f9f80198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37334/0 (socket says 172.21.15.46:37334) 2024-04-26T21:34:46.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.319+0000 7f9f85d02640 1 -- 172.21.15.46:0/910877039 learned_addr learned my addr 172.21.15.46:0/910877039 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:46.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.319+0000 7f9f85501640 1 --2- 172.21.15.46:0/910877039 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f80103bb0 0x7f9f80198fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:46.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.319+0000 7f9f86503640 1 --2- 172.21.15.46:0/910877039 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f80104570 0x7f9f801a0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:46.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.319+0000 7f9f85d02640 1 -- 172.21.15.46:0/910877039 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f80104570 msgr2=0x7f9f801a0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:46.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.319+0000 7f9f85d02640 1 --2- 172.21.15.46:0/910877039 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f80104570 0x7f9f801a0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.319+0000 7f9f85d02640 1 -- 172.21.15.46:0/910877039 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f80103bb0 msgr2=0x7f9f80198fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:46.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.319+0000 7f9f85d02640 1 --2- 172.21.15.46:0/910877039 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f80103bb0 0x7f9f80198fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.319+0000 7f9f85d02640 1 -- 172.21.15.46:0/910877039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f801a06c0 con 0x7f9f801029c0 2024-04-26T21:34:46.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.319+0000 7f9f85d02640 1 --2- 172.21.15.46:0/910877039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f801029c0 0x7f9f80198aa0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c00d770 tx=0x7f9f7c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:46.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.319+0000 7f9f76ffd640 1 -- 172.21.15.46:0/910877039 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c017070 con 0x7f9f801029c0 2024-04-26T21:34:46.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.319+0000 7f9f87f66640 1 -- 172.21.15.46:0/910877039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f801a09b0 con 0x7f9f801029c0 2024-04-26T21:34:46.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.319+0000 7f9f76ffd640 1 -- 172.21.15.46:0/910877039 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f7c009c30 con 0x7f9f801029c0 2024-04-26T21:34:46.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.319+0000 7f9f76ffd640 1 -- 172.21.15.46:0/910877039 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c003cc0 con 0x7f9f801029c0 2024-04-26T21:34:46.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.319+0000 7f9f87f66640 1 -- 172.21.15.46:0/910877039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f801a0f20 con 0x7f9f801029c0 2024-04-26T21:34:46.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.320+0000 7f9f87f66640 1 -- 172.21.15.46:0/910877039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f80109c10 con 0x7f9f801029c0 2024-04-26T21:34:46.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.321+0000 7f9f76ffd640 1 -- 172.21.15.46:0/910877039 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9f7c003e60 con 0x7f9f801029c0 2024-04-26T21:34:46.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.322+0000 7f9f76ffd640 1 --2- 172.21.15.46:0/910877039 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f9f6c0a3cf0 0x7f9f6c0a61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:46.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.322+0000 7f9f76ffd640 1 -- 172.21.15.46:0/910877039 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f9f7c0d02e0 con 0x7f9f801029c0 2024-04-26T21:34:46.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.322+0000 7f9f85501640 1 --2- 172.21.15.46:0/910877039 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f9f6c0a3cf0 0x7f9f6c0a61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:46.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.323+0000 7f9f85501640 1 --2- 172.21.15.46:0/910877039 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f9f6c0a3cf0 0x7f9f6c0a61b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9f70031500 tx=0x7f9f70005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:46.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.326+0000 7f9f76ffd640 1 -- 172.21.15.46:0/910877039 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9f7c009de0 con 0x7f9f801029c0 2024-04-26T21:34:46.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.553+0000 7f9f87f66640 1 -- 172.21.15.46:0/910877039 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f9f8002dad0 con 0x7f9f801029c0 2024-04-26T21:34:46.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.672+0000 7f9f76ffd640 1 -- 172.21.15.46:0/910877039 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v23) ==== 118+0+0 (secure 0 0 0) 0x7f9f7c09c4e0 con 0x7f9f801029c0 2024-04-26T21:34:46.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.674+0000 7f9f87f66640 1 -- 172.21.15.46:0/910877039 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f9f6c0a3cf0 msgr2=0x7f9f6c0a61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:46.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.674+0000 7f9f87f66640 1 --2- 172.21.15.46:0/910877039 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f9f6c0a3cf0 0x7f9f6c0a61b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9f70031500 tx=0x7f9f70005a20 comp rx=0 tx=0).stop 2024-04-26T21:34:46.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.674+0000 7f9f87f66640 1 -- 172.21.15.46:0/910877039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f801029c0 msgr2=0x7f9f80198aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:46.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.674+0000 7f9f87f66640 1 --2- 172.21.15.46:0/910877039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f801029c0 0x7f9f80198aa0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c00d770 tx=0x7f9f7c00dc30 comp rx=0 tx=0).stop 2024-04-26T21:34:46.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.674+0000 7f9f87f66640 1 -- 172.21.15.46:0/910877039 shutdown_connections 2024-04-26T21:34:46.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.674+0000 7f9f87f66640 1 --2- 172.21.15.46:0/910877039 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f9f6c0a3cf0 0x7f9f6c0a61b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.674+0000 7f9f87f66640 1 --2- 172.21.15.46:0/910877039 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f9f80104570 0x7f9f801a0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.674+0000 7f9f87f66640 1 --2- 172.21.15.46:0/910877039 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9f80103bb0 0x7f9f80198fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.674+0000 7f9f87f66640 1 --2- 172.21.15.46:0/910877039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f801029c0 0x7f9f80198aa0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.674+0000 7f9f87f66640 1 -- 172.21.15.46:0/910877039 >> 172.21.15.46:0/910877039 conn(0x7f9f800fdf50 msgr2=0x7f9f80106de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:46.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.674+0000 7f9f87f66640 1 -- 172.21.15.46:0/910877039 shutdown_connections 2024-04-26T21:34:46.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.674+0000 7f9f87f66640 1 -- 172.21.15.46:0/910877039 wait complete. 2024-04-26T21:34:46.686 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-04-26T21:34:46.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.790+0000 7efc6b847640 1 -- 172.21.15.46:0/720590527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc641034f0 msgr2=0x7efc64109d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:46.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.790+0000 7efc6b847640 1 --2- 172.21.15.46:0/720590527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc641034f0 0x7efc64109d80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7efc54009ca0 tx=0x7efc5402a8c0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.790+0000 7efc6b847640 1 -- 172.21.15.46:0/720590527 shutdown_connections 2024-04-26T21:34:46.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.790+0000 7efc6b847640 1 --2- 172.21.15.46:0/720590527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc641034f0 0x7efc64109d80 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.790+0000 7efc6b847640 1 --2- 172.21.15.46:0/720590527 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc64102b30 0x7efc64102fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.790+0000 7efc6b847640 1 --2- 172.21.15.46:0/720590527 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc64101940 0x7efc64101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.790+0000 7efc6b847640 1 -- 172.21.15.46:0/720590527 >> 172.21.15.46:0/720590527 conn(0x7efc640fceb0 msgr2=0x7efc640ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:46.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.790+0000 7efc6b847640 1 -- 172.21.15.46:0/720590527 shutdown_connections 2024-04-26T21:34:46.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.790+0000 7efc6b847640 1 -- 172.21.15.46:0/720590527 wait complete. 2024-04-26T21:34:46.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.791+0000 7efc6b847640 1 Processor -- start 2024-04-26T21:34:46.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.791+0000 7efc6b847640 1 -- start start 2024-04-26T21:34:46.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.791+0000 7efc6b847640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc64101940 0x7efc6410c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:46.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.791+0000 7efc6b847640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc64102b30 0x7efc6410ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:46.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.791+0000 7efc6b847640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc641034f0 0x7efc6410b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:46.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.791+0000 7efc6b847640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7efc6410e4d0 con 0x7efc64102b30 2024-04-26T21:34:46.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.791+0000 7efc6b847640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7efc6410e350 con 0x7efc641034f0 2024-04-26T21:34:46.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.791+0000 7efc6b847640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7efc6410e650 con 0x7efc64101940 2024-04-26T21:34:46.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.791+0000 7efc68de2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc64102b30 0x7efc6410ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:46.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.791+0000 7efc68de2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc64102b30 0x7efc6410ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37352/0 (socket says 172.21.15.46:37352) 2024-04-26T21:34:46.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.791+0000 7efc68de2640 1 -- 172.21.15.46:0/123926620 learned_addr learned my addr 172.21.15.46:0/123926620 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:34:46.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.791+0000 7efc69de4640 1 --2- 172.21.15.46:0/123926620 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc641034f0 0x7efc6410b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:46.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.791+0000 7efc695e3640 1 --2- 172.21.15.46:0/123926620 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc64101940 0x7efc6410c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:46.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.791+0000 7efc68de2640 1 -- 172.21.15.46:0/123926620 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc64101940 msgr2=0x7efc6410c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:46.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.791+0000 7efc68de2640 1 --2- 172.21.15.46:0/123926620 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7efc64101940 0x7efc6410c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.791+0000 7efc68de2640 1 -- 172.21.15.46:0/123926620 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc641034f0 msgr2=0x7efc6410b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:46.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.791+0000 7efc68de2640 1 --2- 172.21.15.46:0/123926620 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7efc641034f0 0x7efc6410b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.791+0000 7efc68de2640 1 -- 172.21.15.46:0/123926620 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc6410b990 con 0x7efc64102b30 2024-04-26T21:34:46.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.792+0000 7efc68de2640 1 --2- 172.21.15.46:0/123926620 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc64102b30 0x7efc6410ab90 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7efc5800c920 tx=0x7efc5800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:46.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.792+0000 7efc627fc640 1 -- 172.21.15.46:0/123926620 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc58007c70 con 0x7efc64102b30 2024-04-26T21:34:46.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.792+0000 7efc6b847640 1 -- 172.21.15.46:0/123926620 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc641a2c40 con 0x7efc64102b30 2024-04-26T21:34:46.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.792+0000 7efc627fc640 1 -- 172.21.15.46:0/123926620 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc58007e10 con 0x7efc64102b30 2024-04-26T21:34:46.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.792+0000 7efc627fc640 1 -- 172.21.15.46:0/123926620 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc58018820 con 0x7efc64102b30 2024-04-26T21:34:46.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.792+0000 7efc6b847640 1 -- 172.21.15.46:0/123926620 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc641a3160 con 0x7efc64102b30 2024-04-26T21:34:46.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.793+0000 7efc6b847640 1 -- 172.21.15.46:0/123926620 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc64108b90 con 0x7efc64102b30 2024-04-26T21:34:46.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.794+0000 7efc627fc640 1 -- 172.21.15.46:0/123926620 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7efc5800f450 con 0x7efc64102b30 2024-04-26T21:34:46.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.795+0000 7efc627fc640 1 --2- 172.21.15.46:0/123926620 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7efc4c0a3cf0 0x7efc4c0a61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:46.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.795+0000 7efc695e3640 1 --2- 172.21.15.46:0/123926620 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7efc4c0a3cf0 0x7efc4c0a61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:46.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.795+0000 7efc627fc640 1 -- 172.21.15.46:0/123926620 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7efc580d05e0 con 0x7efc64102b30 2024-04-26T21:34:46.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.795+0000 7efc695e3640 1 --2- 172.21.15.46:0/123926620 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7efc4c0a3cf0 0x7efc4c0a61b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7efc64102990 tx=0x7efc5c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:46.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:46.800+0000 7efc627fc640 1 -- 172.21.15.46:0/123926620 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7efc58014030 con 0x7efc64102b30 2024-04-26T21:34:47.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:34:47.015+0000 7efc6b847640 1 -- 172.21.15.46:0/123926620 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7efc64062ed0 con 0x7efc4c0a3cf0 2024-04-26T21:34:47.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:46 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/910877039' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-26T21:34:47.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:46 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/910877039' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-26T21:34:47.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:46 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/910877039' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-26T21:34:48.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:47 smithi142 ceph-mon[32944]: pgmap v355: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:34:48.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:47 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/910877039' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-26T21:34:48.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:47 smithi142 ceph-mon[32944]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:34:48.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:47 smithi046 ceph-mon[29201]: pgmap v355: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:34:48.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:47 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/910877039' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-26T21:34:48.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:47 smithi046 ceph-mon[29201]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:34:48.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:47 smithi050 ceph-mon[32753]: pgmap v355: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:34:48.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:47 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/910877039' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-26T21:34:48.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:47 smithi050 ceph-mon[32753]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:34:49.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:48 smithi142 ceph-mon[32944]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:34:49.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:48 smithi046 ceph-mon[29201]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:34:49.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:48 smithi050 ceph-mon[32753]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:34:50.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:49 smithi142 ceph-mon[32944]: pgmap v356: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:50.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:49 smithi046 ceph-mon[29201]: pgmap v356: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:50.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:49 smithi050 ceph-mon[32753]: pgmap v356: 97 pgs: 97 active+clean; 629 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:34:52.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:51 smithi142 ceph-mon[32944]: pgmap v357: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 767 B/s wr, 8 op/s 2024-04-26T21:34:52.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:51 smithi046 ceph-mon[29201]: pgmap v357: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 767 B/s wr, 8 op/s 2024-04-26T21:34:52.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:51 smithi050 ceph-mon[32753]: pgmap v357: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 767 B/s wr, 8 op/s 2024-04-26T21:34:54.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:53 smithi142 ceph-mon[32944]: pgmap v358: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-26T21:34:54.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:53 smithi046 ceph-mon[29201]: pgmap v358: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-26T21:34:54.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:53 smithi050 ceph-mon[32753]: pgmap v358: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-26T21:34:56.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:55 smithi142 ceph-mon[32944]: pgmap v359: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-26T21:34:56.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:55 smithi046 ceph-mon[29201]: pgmap v359: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-26T21:34:56.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:55 smithi050 ceph-mon[32753]: pgmap v359: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-26T21:34:58.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:57 smithi142 ceph-mon[32944]: pgmap v360: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 853 B/s wr, 8 op/s 2024-04-26T21:34:58.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:57 smithi046 ceph-mon[29201]: pgmap v360: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 853 B/s wr, 8 op/s 2024-04-26T21:34:58.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:57 smithi050 ceph-mon[32753]: pgmap v360: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 853 B/s wr, 8 op/s 2024-04-26T21:35:00.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:34:59 smithi142 ceph-mon[32944]: pgmap v361: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-26T21:35:00.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:34:59 smithi046 ceph-mon[29201]: pgmap v361: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-26T21:35:00.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:34:59 smithi050 ceph-mon[32753]: pgmap v361: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-26T21:35:02.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:01 smithi142 ceph-mon[32944]: pgmap v362: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 853 B/s wr, 8 op/s 2024-04-26T21:35:02.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:01 smithi046 ceph-mon[29201]: pgmap v362: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 853 B/s wr, 8 op/s 2024-04-26T21:35:02.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:01 smithi050 ceph-mon[32753]: pgmap v362: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 853 B/s wr, 8 op/s 2024-04-26T21:35:04.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:03 smithi142 ceph-mon[32944]: pgmap v363: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:35:04.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:03 smithi046 ceph-mon[29201]: pgmap v363: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:35:04.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:03 smithi050 ceph-mon[32753]: pgmap v363: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:35:04.982 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:04 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:35:05.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:04 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:35:05.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:04 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:35:06.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:05 smithi142 ceph-mon[32944]: pgmap v364: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:06.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:05 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:35:06.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:05 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:35:06.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:05 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:35:06.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:05 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:35:06.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:05 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:35:06.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:05 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:35:06.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:05 smithi046 ceph-mon[29201]: pgmap v364: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:06.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:05 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:35:06.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:05 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:35:06.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:05 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:35:06.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:05 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:35:06.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:05 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:35:06.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:05 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:35:06.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:05 smithi050 ceph-mon[32753]: pgmap v364: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:06.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:05 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:35:06.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:05 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:35:06.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:05 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:35:06.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:05 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:35:06.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:05 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:35:06.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:05 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:35:07.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:06 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:35:07.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:06 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:35:07.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:06 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:35:07.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:06 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:35:07.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:06 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:35:07.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:06 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:35:07.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:06 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:35:07.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:06 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:35:07.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:06 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:35:08.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:07 smithi142 ceph-mon[32944]: pgmap v365: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:08.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:07 smithi046 ceph-mon[29201]: pgmap v365: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:08.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:07 smithi050 ceph-mon[32753]: pgmap v365: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:10.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:09 smithi142 ceph-mon[32944]: pgmap v366: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:10.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:09 smithi046 ceph-mon[29201]: pgmap v366: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:10.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:09 smithi050 ceph-mon[32753]: pgmap v366: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:12.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:11 smithi142 ceph-mon[32944]: pgmap v367: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:12.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:11 smithi046 ceph-mon[29201]: pgmap v367: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:12.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:11 smithi050 ceph-mon[32753]: pgmap v367: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:14.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:13 smithi142 ceph-mon[32944]: pgmap v368: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:14.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:13 smithi046 ceph-mon[29201]: pgmap v368: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:14.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:13 smithi050 ceph-mon[32753]: pgmap v368: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:16.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:15 smithi142 ceph-mon[32944]: pgmap v369: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:16.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:15 smithi046 ceph-mon[29201]: pgmap v369: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:16.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:15 smithi050 ceph-mon[32753]: pgmap v369: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:18.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:17 smithi142 ceph-mon[32944]: pgmap v370: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:18.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:17 smithi046 ceph-mon[29201]: pgmap v370: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:18.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:17 smithi050 ceph-mon[32753]: pgmap v370: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:20.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:19 smithi142 ceph-mon[32944]: pgmap v371: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:20.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:19 smithi046 ceph-mon[29201]: pgmap v371: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:20.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:19 smithi050 ceph-mon[32753]: pgmap v371: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:22.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:21 smithi142 ceph-mon[32944]: pgmap v372: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:22.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:21 smithi046 ceph-mon[29201]: pgmap v372: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:22.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:21 smithi050 ceph-mon[32753]: pgmap v372: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:24.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:23 smithi142 ceph-mon[32944]: pgmap v373: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:24.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:23 smithi046 ceph-mon[29201]: pgmap v373: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:24.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:23 smithi050 ceph-mon[32753]: pgmap v373: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:26.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:25 smithi142 ceph-mon[32944]: pgmap v374: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:26.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:25 smithi046 ceph-mon[29201]: pgmap v374: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:26.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:25 smithi050 ceph-mon[32753]: pgmap v374: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:28.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:27 smithi142 ceph-mon[32944]: pgmap v375: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:28.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:27 smithi046 ceph-mon[29201]: pgmap v375: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:28.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:27 smithi050 ceph-mon[32753]: pgmap v375: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:30.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:29 smithi142 ceph-mon[32944]: pgmap v376: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:30.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:29 smithi046 ceph-mon[29201]: pgmap v376: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:30.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:29 smithi050 ceph-mon[32753]: pgmap v376: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:32.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:31 smithi142 ceph-mon[32944]: pgmap v377: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:32.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:31 smithi046 ceph-mon[29201]: pgmap v377: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:32.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:31 smithi050 ceph-mon[32753]: pgmap v377: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:34.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:33 smithi142 ceph-mon[32944]: pgmap v378: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:34.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:33 smithi046 ceph-mon[29201]: pgmap v378: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:34.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:33 smithi050 ceph-mon[32753]: pgmap v378: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:36.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:35 smithi142 ceph-mon[32944]: pgmap v379: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:36.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:35 smithi046 ceph-mon[29201]: pgmap v379: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:36.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:35 smithi050 ceph-mon[32753]: pgmap v379: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:38.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:37 smithi142 ceph-mon[32944]: pgmap v380: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:38.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:37 smithi046 ceph-mon[29201]: pgmap v380: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:38.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:37 smithi050 ceph-mon[32753]: pgmap v380: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:40.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:39 smithi142 ceph-mon[32944]: pgmap v381: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:40.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:39 smithi046 ceph-mon[29201]: pgmap v381: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:40.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:39 smithi050 ceph-mon[32753]: pgmap v381: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:42.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:41 smithi142 ceph-mon[32944]: pgmap v382: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:42.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:41 smithi046 ceph-mon[29201]: pgmap v382: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:42.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:41 smithi050 ceph-mon[32753]: pgmap v382: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:44.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:43 smithi142 ceph-mon[32944]: pgmap v383: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:44.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:43 smithi046 ceph-mon[29201]: pgmap v383: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:44.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:43 smithi050 ceph-mon[32753]: pgmap v383: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:46.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:45 smithi142 ceph-mon[32944]: pgmap v384: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:46.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:45 smithi046 ceph-mon[29201]: pgmap v384: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:46.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:45 smithi050 ceph-mon[32753]: pgmap v384: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:48.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:47 smithi142 ceph-mon[32944]: pgmap v385: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:48.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:47 smithi046 ceph-mon[29201]: pgmap v385: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:48.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:47 smithi050 ceph-mon[32753]: pgmap v385: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:50.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:49 smithi142 ceph-mon[32944]: pgmap v386: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:50.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:49 smithi046 ceph-mon[29201]: pgmap v386: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:50.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:49 smithi050 ceph-mon[32753]: pgmap v386: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:52.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:51 smithi142 ceph-mon[32944]: pgmap v387: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:52.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:51 smithi046 ceph-mon[29201]: pgmap v387: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:52.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:51 smithi050 ceph-mon[32753]: pgmap v387: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:54.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:53 smithi142 ceph-mon[32944]: pgmap v388: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:54.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:53 smithi046 ceph-mon[29201]: pgmap v388: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:54.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:53 smithi050 ceph-mon[32753]: pgmap v388: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:56.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:55 smithi142 ceph-mon[32944]: pgmap v389: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:56.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:55 smithi046 ceph-mon[29201]: pgmap v389: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:56.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:55 smithi050 ceph-mon[32753]: pgmap v389: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:35:58.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:57 smithi142 ceph-mon[32944]: pgmap v390: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:58.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:57 smithi046 ceph-mon[29201]: pgmap v390: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:35:58.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:57 smithi050 ceph-mon[32753]: pgmap v390: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:00.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:35:59 smithi142 ceph-mon[32944]: pgmap v391: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:36:00.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:35:59 smithi046 ceph-mon[29201]: pgmap v391: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:36:00.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:35:59 smithi050 ceph-mon[32753]: pgmap v391: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:36:02.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:01 smithi046 ceph-mon[29201]: pgmap v392: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T21:36:02.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:01 smithi050 ceph-mon[32753]: pgmap v392: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T21:36:02.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:01 smithi142 ceph-mon[32944]: pgmap v392: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T21:36:04.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:03 smithi046 ceph-mon[29201]: pgmap v393: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:36:04.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:03 smithi050 ceph-mon[32753]: pgmap v393: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:36:04.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:03 smithi142 ceph-mon[32944]: pgmap v393: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:36:06.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:05 smithi046 ceph-mon[29201]: pgmap v394: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:36:06.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:05 smithi050 ceph-mon[32753]: pgmap v394: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:36:06.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:05 smithi142 ceph-mon[32944]: pgmap v394: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:36:07.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:06 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:36:07.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:06 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:36:07.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:06 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:36:08.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:07 smithi046 ceph-mon[29201]: pgmap v395: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T21:36:08.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:07 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:36:08.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:07 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:36:08.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:07 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:36:08.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:07 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:36:08.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:07 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:36:08.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:07 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:36:08.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:07 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:36:08.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:07 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:36:08.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:07 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:36:08.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:07 smithi050 ceph-mon[32753]: pgmap v395: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T21:36:08.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:07 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:36:08.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:07 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:36:08.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:07 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:36:08.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:07 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:36:08.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:07 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:36:08.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:07 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:36:08.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:07 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:36:08.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:07 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:36:08.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:07 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:36:08.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:07 smithi142 ceph-mon[32944]: pgmap v395: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T21:36:08.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:07 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:36:08.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:07 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:36:08.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:07 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:36:08.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:07 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:36:08.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:07 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:36:08.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:07 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:36:08.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:07 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:36:08.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:07 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:36:08.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:07 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:36:10.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:09 smithi050 ceph-mon[32753]: pgmap v396: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:36:10.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:09 smithi142 ceph-mon[32944]: pgmap v396: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:36:10.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:09 smithi046 ceph-mon[29201]: pgmap v396: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:36:12.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:11 smithi050 ceph-mon[32753]: pgmap v397: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:12.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:11 smithi142 ceph-mon[32944]: pgmap v397: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:12.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:11 smithi046 ceph-mon[29201]: pgmap v397: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:14.093 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:13 smithi142 ceph-mon[32944]: pgmap v398: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:14.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:13 smithi050 ceph-mon[32753]: pgmap v398: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:14.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:13 smithi046 ceph-mon[29201]: pgmap v398: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:16.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:15 smithi050 ceph-mon[32753]: pgmap v399: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:16.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:15 smithi142 ceph-mon[32944]: pgmap v399: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:16.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:15 smithi046 ceph-mon[29201]: pgmap v399: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:18.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:17 smithi050 ceph-mon[32753]: pgmap v400: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:18.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:17 smithi142 ceph-mon[32944]: pgmap v400: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:18.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:17 smithi046 ceph-mon[29201]: pgmap v400: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:20.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:19 smithi050 ceph-mon[32753]: pgmap v401: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:20.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:19 smithi142 ceph-mon[32944]: pgmap v401: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:20.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:19 smithi046 ceph-mon[29201]: pgmap v401: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:22.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:21 smithi050 ceph-mon[32753]: pgmap v402: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:22.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:21 smithi142 ceph-mon[32944]: pgmap v402: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:22.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:21 smithi046 ceph-mon[29201]: pgmap v402: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:24.105 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:23 smithi046 ceph-mon[29201]: pgmap v403: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:24.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:23 smithi050 ceph-mon[32753]: pgmap v403: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:24.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:23 smithi142 ceph-mon[32944]: pgmap v403: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:26.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:25 smithi050 ceph-mon[32753]: pgmap v404: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:26.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:25 smithi142 ceph-mon[32944]: pgmap v404: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:26.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:25 smithi046 ceph-mon[29201]: pgmap v404: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:28.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:27 smithi050 ceph-mon[32753]: pgmap v405: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:28.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:27 smithi142 ceph-mon[32944]: pgmap v405: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:28.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:27 smithi046 ceph-mon[29201]: pgmap v405: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:30.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:29 smithi050 ceph-mon[32753]: pgmap v406: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:36:30.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:29 smithi142 ceph-mon[32944]: pgmap v406: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:36:30.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:29 smithi046 ceph-mon[29201]: pgmap v406: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:36:32.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:31 smithi050 ceph-mon[32753]: pgmap v407: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:32.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:31 smithi142 ceph-mon[32944]: pgmap v407: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:32.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:31 smithi046 ceph-mon[29201]: pgmap v407: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:34.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:33 smithi050 ceph-mon[32753]: pgmap v408: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:36:34.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:33 smithi142 ceph-mon[32944]: pgmap v408: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:36:34.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:33 smithi046 ceph-mon[29201]: pgmap v408: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:36:36.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:35 smithi050 ceph-mon[32753]: pgmap v409: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:36:36.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:35 smithi142 ceph-mon[32944]: pgmap v409: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:36:36.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:35 smithi046 ceph-mon[29201]: pgmap v409: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:36:38.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:37 smithi050 ceph-mon[32753]: pgmap v410: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:38.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:37 smithi142 ceph-mon[32944]: pgmap v410: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:38.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:37 smithi046 ceph-mon[29201]: pgmap v410: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:40.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:39 smithi050 ceph-mon[32753]: pgmap v411: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:40.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:39 smithi142 ceph-mon[32944]: pgmap v411: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:40.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:39 smithi046 ceph-mon[29201]: pgmap v411: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:42.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:41 smithi050 ceph-mon[32753]: pgmap v412: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:42.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:41 smithi142 ceph-mon[32944]: pgmap v412: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:42.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:41 smithi046 ceph-mon[29201]: pgmap v412: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:44.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:43 smithi050 ceph-mon[32753]: pgmap v413: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:44.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:43 smithi142 ceph-mon[32944]: pgmap v413: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:44.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:43 smithi046 ceph-mon[29201]: pgmap v413: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:46.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:45 smithi050 ceph-mon[32753]: pgmap v414: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:46.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:45 smithi142 ceph-mon[32944]: pgmap v414: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:46.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:45 smithi046 ceph-mon[29201]: pgmap v414: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:48.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:47 smithi050 ceph-mon[32753]: pgmap v415: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:48.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:47 smithi142 ceph-mon[32944]: pgmap v415: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:48.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:47 smithi046 ceph-mon[29201]: pgmap v415: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:50.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:49 smithi142 ceph-mon[32944]: pgmap v416: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:50.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:49 smithi046 ceph-mon[29201]: pgmap v416: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:50.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:49 smithi050 ceph-mon[32753]: pgmap v416: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:52.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:51 smithi142 ceph-mon[32944]: pgmap v417: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:52.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:51 smithi046 ceph-mon[29201]: pgmap v417: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:52.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:51 smithi050 ceph-mon[32753]: pgmap v417: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:54.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:53 smithi142 ceph-mon[32944]: pgmap v418: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:54.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:53 smithi046 ceph-mon[29201]: pgmap v418: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:54.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:53 smithi050 ceph-mon[32753]: pgmap v418: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:56.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:55 smithi142 ceph-mon[32944]: pgmap v419: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:56.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:55 smithi046 ceph-mon[29201]: pgmap v419: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:56.370 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:55 smithi050 ceph-mon[32753]: pgmap v419: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:36:58.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:57 smithi142 ceph-mon[32944]: pgmap v420: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:58.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:57 smithi046 ceph-mon[29201]: pgmap v420: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:36:58.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:57 smithi050 ceph-mon[32753]: pgmap v420: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:00.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:36:59 smithi142 ceph-mon[32944]: pgmap v421: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:00.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:36:59 smithi046 ceph-mon[29201]: pgmap v421: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:00.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:36:59 smithi050 ceph-mon[32753]: pgmap v421: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:01.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:00 smithi142 ceph-mon[32944]: pgmap v422: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:01.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:00 smithi046 ceph-mon[29201]: pgmap v422: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:01.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:00 smithi050 ceph-mon[32753]: pgmap v422: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:03.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:03 smithi050 ceph-mon[32753]: pgmap v423: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:03.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:03 smithi142 ceph-mon[32944]: pgmap v423: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:03.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:03 smithi046 ceph-mon[29201]: pgmap v423: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:05.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:05 smithi050 ceph-mon[32753]: pgmap v424: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:05.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:05 smithi142 ceph-mon[32944]: pgmap v424: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:05.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:05 smithi046 ceph-mon[29201]: pgmap v424: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:07.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:07 smithi050 ceph-mon[32753]: pgmap v425: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:07.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:07 smithi142 ceph-mon[32944]: pgmap v425: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:07.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:07 smithi046 ceph-mon[29201]: pgmap v425: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:08.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:08 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:37:08.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:08 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:37:08.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:08 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:37:09.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:09 smithi050 ceph-mon[32753]: pgmap v426: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:09.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:09 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:37:09.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:09 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:37:09.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:09 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:37:09.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:09 smithi142 ceph-mon[32944]: pgmap v426: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:09.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:09 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:37:09.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:09 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:37:09.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:09 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:37:09.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:09 smithi046 ceph-mon[29201]: pgmap v426: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:09.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:09 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:37:09.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:09 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:37:09.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:09 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:37:11.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:11 smithi050 ceph-mon[32753]: pgmap v427: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:11.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:11 smithi142 ceph-mon[32944]: pgmap v427: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:11.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:11 smithi046 ceph-mon[29201]: pgmap v427: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:13.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:13 smithi050 ceph-mon[32753]: pgmap v428: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:13.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:13 smithi142 ceph-mon[32944]: pgmap v428: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:13.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:13 smithi046 ceph-mon[29201]: pgmap v428: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:15.343 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:15 smithi142 ceph-mon[32944]: pgmap v429: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:15.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:15 smithi050 ceph-mon[32753]: pgmap v429: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:15.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:15 smithi046 ceph-mon[29201]: pgmap v429: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:17.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:17 smithi050 ceph-mon[32753]: pgmap v430: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:17.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:17 smithi142 ceph-mon[32944]: pgmap v430: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:17.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:17 smithi046 ceph-mon[29201]: pgmap v430: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:19.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:19 smithi050 ceph-mon[32753]: pgmap v431: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:19.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:19 smithi142 ceph-mon[32944]: pgmap v431: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:19.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:19 smithi046 ceph-mon[29201]: pgmap v431: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:21.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:21 smithi050 ceph-mon[32753]: pgmap v432: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:21.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:21 smithi142 ceph-mon[32944]: pgmap v432: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:21.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:21 smithi046 ceph-mon[29201]: pgmap v432: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:23.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:23 smithi050 ceph-mon[32753]: pgmap v433: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:23.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:23 smithi142 ceph-mon[32944]: pgmap v433: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:23.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:23 smithi046 ceph-mon[29201]: pgmap v433: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:25.355 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:25 smithi046 ceph-mon[29201]: pgmap v434: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:25.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:25 smithi050 ceph-mon[32753]: pgmap v434: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:25.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:25 smithi142 ceph-mon[32944]: pgmap v434: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:27.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:27 smithi050 ceph-mon[32753]: pgmap v435: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:27.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:27 smithi142 ceph-mon[32944]: pgmap v435: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:27.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:27 smithi046 ceph-mon[29201]: pgmap v435: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:29.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:29 smithi050 ceph-mon[32753]: pgmap v436: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:29.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:29 smithi142 ceph-mon[32944]: pgmap v436: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:29.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:29 smithi046 ceph-mon[29201]: pgmap v436: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:31.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:31 smithi050 ceph-mon[32753]: pgmap v437: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:31.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:31 smithi142 ceph-mon[32944]: pgmap v437: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:31.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:31 smithi046 ceph-mon[29201]: pgmap v437: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:33.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:33 smithi050 ceph-mon[32753]: pgmap v438: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:33.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:33 smithi142 ceph-mon[32944]: pgmap v438: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:33.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:33 smithi046 ceph-mon[29201]: pgmap v438: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:35.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:35 smithi050 ceph-mon[32753]: pgmap v439: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:35.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:35 smithi142 ceph-mon[32944]: pgmap v439: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:35.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:35 smithi046 ceph-mon[29201]: pgmap v439: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:37.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:37 smithi050 ceph-mon[32753]: pgmap v440: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:37.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:37 smithi142 ceph-mon[32944]: pgmap v440: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:37.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:37 smithi046 ceph-mon[29201]: pgmap v440: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:39.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:39 smithi050 ceph-mon[32753]: pgmap v441: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:39.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:39 smithi142 ceph-mon[32944]: pgmap v441: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:39.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:39 smithi046 ceph-mon[29201]: pgmap v441: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:41.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:41 smithi050 ceph-mon[32753]: pgmap v442: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:41.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:41 smithi142 ceph-mon[32944]: pgmap v442: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:41.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:41 smithi046 ceph-mon[29201]: pgmap v442: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:43.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:43 smithi142 ceph-mon[32944]: pgmap v443: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:43.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:43 smithi046 ceph-mon[29201]: pgmap v443: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:43.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:43 smithi050 ceph-mon[32753]: pgmap v443: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:45.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:45 smithi142 ceph-mon[32944]: pgmap v444: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:45.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:45 smithi046 ceph-mon[29201]: pgmap v444: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:45.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:45 smithi050 ceph-mon[32753]: pgmap v444: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:46.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:37:46.817+0000 7efc617fa640 1 -- 172.21.15.46:0/123926620 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc440034c0 con 0x7efc64102b30 2024-04-26T21:37:46.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:37:46.818+0000 7efc627fc640 1 -- 172.21.15.46:0/123926620 <== mon.0 v2:172.21.15.46:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc440034c0 con 0x7efc64102b30 2024-04-26T21:37:47.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:47 smithi142 ceph-mon[32944]: pgmap v445: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:47.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:47 smithi046 ceph-mon[29201]: pgmap v445: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:47.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:47 smithi050 ceph-mon[32753]: pgmap v445: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:49.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:49 smithi142 ceph-mon[32944]: pgmap v446: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:49.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:49 smithi046 ceph-mon[29201]: pgmap v446: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:49.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:49 smithi050 ceph-mon[32753]: pgmap v446: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:51.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:51 smithi142 ceph-mon[32944]: pgmap v447: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:51.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:51 smithi046 ceph-mon[29201]: pgmap v447: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:51.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:51 smithi050 ceph-mon[32753]: pgmap v447: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:53.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:53 smithi142 ceph-mon[32944]: pgmap v448: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:53.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:53 smithi046 ceph-mon[29201]: pgmap v448: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:53.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:53 smithi050 ceph-mon[32753]: pgmap v448: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:55.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:55 smithi142 ceph-mon[32944]: pgmap v449: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:55.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:55 smithi046 ceph-mon[29201]: pgmap v449: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:55.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:55 smithi050 ceph-mon[32753]: pgmap v449: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:57.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:57 smithi142 ceph-mon[32944]: pgmap v450: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:57.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:57 smithi046 ceph-mon[29201]: pgmap v450: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:57.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:57 smithi050 ceph-mon[32753]: pgmap v450: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:37:59.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:37:59 smithi142 ceph-mon[32944]: pgmap v451: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:59.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:37:59 smithi046 ceph-mon[29201]: pgmap v451: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:37:59.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:37:59 smithi050 ceph-mon[32753]: pgmap v451: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:01.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:01 smithi142 ceph-mon[32944]: pgmap v452: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:01.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:01 smithi046 ceph-mon[29201]: pgmap v452: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:01.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:01 smithi050 ceph-mon[32753]: pgmap v452: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:03.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:03 smithi142 ceph-mon[32944]: pgmap v453: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:03.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:03 smithi046 ceph-mon[29201]: pgmap v453: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:03.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:03 smithi050 ceph-mon[32753]: pgmap v453: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:05.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:05 smithi142 ceph-mon[32944]: pgmap v454: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:05.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:05 smithi046 ceph-mon[29201]: pgmap v454: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:05.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:05 smithi050 ceph-mon[32753]: pgmap v454: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:07.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:07 smithi142 ceph-mon[32944]: pgmap v455: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:07.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:07 smithi046 ceph-mon[29201]: pgmap v455: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:07.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:07 smithi050 ceph-mon[32753]: pgmap v455: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:09.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:09 smithi142 ceph-mon[32944]: pgmap v456: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:09.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:09 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:38:09.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:09 smithi046 ceph-mon[29201]: pgmap v456: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:09.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:09 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:38:09.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:09 smithi050 ceph-mon[32753]: pgmap v456: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:09.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:09 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:38:10.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:10 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:38:10.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:10 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:38:10.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:10 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:38:10.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:10 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:38:10.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:10 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:38:10.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:10 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:38:10.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:10 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:38:10.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:10 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:38:10.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:10 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:38:11.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:11 smithi050 ceph-mon[32753]: pgmap v457: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:11.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:11 smithi142 ceph-mon[32944]: pgmap v457: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:11.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:11 smithi046 ceph-mon[29201]: pgmap v457: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:13.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:13 smithi050 ceph-mon[32753]: pgmap v458: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:13.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:13 smithi142 ceph-mon[32944]: pgmap v458: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:13.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:13 smithi046 ceph-mon[29201]: pgmap v458: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:15.593 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:15 smithi142 ceph-mon[32944]: pgmap v459: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:15.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:15 smithi050 ceph-mon[32753]: pgmap v459: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:15.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:15 smithi046 ceph-mon[29201]: pgmap v459: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:17.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:17 smithi050 ceph-mon[32753]: pgmap v460: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:17.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:17 smithi142 ceph-mon[32944]: pgmap v460: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:17.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:17 smithi046 ceph-mon[29201]: pgmap v460: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:19.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:19 smithi050 ceph-mon[32753]: pgmap v461: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:19.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:19 smithi142 ceph-mon[32944]: pgmap v461: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:19.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:19 smithi046 ceph-mon[29201]: pgmap v461: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:21.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:21 smithi050 ceph-mon[32753]: pgmap v462: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:21.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:21 smithi142 ceph-mon[32944]: pgmap v462: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:21.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:21 smithi046 ceph-mon[29201]: pgmap v462: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:23.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:23 smithi050 ceph-mon[32753]: pgmap v463: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:23.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:23 smithi142 ceph-mon[32944]: pgmap v463: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:23.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:23 smithi046 ceph-mon[29201]: pgmap v463: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:25.605 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:25 smithi046 ceph-mon[29201]: pgmap v464: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:25.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:25 smithi050 ceph-mon[32753]: pgmap v464: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:25.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:25 smithi142 ceph-mon[32944]: pgmap v464: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:27.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:27 smithi050 ceph-mon[32753]: pgmap v465: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:27.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:27 smithi142 ceph-mon[32944]: pgmap v465: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:27.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:27 smithi046 ceph-mon[29201]: pgmap v465: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:29.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:29 smithi050 ceph-mon[32753]: pgmap v466: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:29.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:29 smithi142 ceph-mon[32944]: pgmap v466: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:29.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:29 smithi046 ceph-mon[29201]: pgmap v466: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:31.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:31 smithi050 ceph-mon[32753]: pgmap v467: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:31.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:31 smithi142 ceph-mon[32944]: pgmap v467: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:31.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:31 smithi046 ceph-mon[29201]: pgmap v467: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:33.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:33 smithi050 ceph-mon[32753]: pgmap v468: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:33.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:33 smithi142 ceph-mon[32944]: pgmap v468: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:33.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:33 smithi046 ceph-mon[29201]: pgmap v468: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:35.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:35 smithi050 ceph-mon[32753]: pgmap v469: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:35.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:35 smithi142 ceph-mon[32944]: pgmap v469: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:35.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:35 smithi046 ceph-mon[29201]: pgmap v469: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:37.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:37 smithi050 ceph-mon[32753]: pgmap v470: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:37.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:37 smithi142 ceph-mon[32944]: pgmap v470: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:37.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:37 smithi046 ceph-mon[29201]: pgmap v470: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:39.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:39 smithi050 ceph-mon[32753]: pgmap v471: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:39.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:39 smithi142 ceph-mon[32944]: pgmap v471: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:39.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:39 smithi046 ceph-mon[29201]: pgmap v471: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:41.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:41 smithi050 ceph-mon[32753]: pgmap v472: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:41.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:41 smithi142 ceph-mon[32944]: pgmap v472: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:41.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:41 smithi046 ceph-mon[29201]: pgmap v472: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:43.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:43 smithi050 ceph-mon[32753]: pgmap v473: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:43.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:43 smithi142 ceph-mon[32944]: pgmap v473: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:43.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:43 smithi046 ceph-mon[29201]: pgmap v473: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:45.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:45 smithi050 ceph-mon[32753]: pgmap v474: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:45.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:45 smithi142 ceph-mon[32944]: pgmap v474: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:45.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:45 smithi046 ceph-mon[29201]: pgmap v474: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:47.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:47 smithi050 ceph-mon[32753]: pgmap v475: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:47.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:47 smithi142 ceph-mon[32944]: pgmap v475: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:47.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:47 smithi046 ceph-mon[29201]: pgmap v475: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:49.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:49 smithi050 ceph-mon[32753]: pgmap v476: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:49.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:49 smithi142 ceph-mon[32944]: pgmap v476: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:49.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:49 smithi046 ceph-mon[29201]: pgmap v476: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:51.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:51 smithi050 ceph-mon[32753]: pgmap v477: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:51.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:51 smithi142 ceph-mon[32944]: pgmap v477: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:51.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:51 smithi046 ceph-mon[29201]: pgmap v477: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:53.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:53 smithi050 ceph-mon[32753]: pgmap v478: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:53.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:53 smithi142 ceph-mon[32944]: pgmap v478: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:53.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:53 smithi046 ceph-mon[29201]: pgmap v478: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:55.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:55 smithi142 ceph-mon[32944]: pgmap v479: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:55.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:55 smithi046 ceph-mon[29201]: pgmap v479: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:55.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:55 smithi050 ceph-mon[32753]: pgmap v479: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:57.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:57 smithi142 ceph-mon[32944]: pgmap v480: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:57.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:57 smithi046 ceph-mon[29201]: pgmap v480: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:57.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:57 smithi050 ceph-mon[32753]: pgmap v480: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:38:59.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:38:59 smithi142 ceph-mon[32944]: pgmap v481: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:59.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:38:59 smithi046 ceph-mon[29201]: pgmap v481: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:38:59.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:38:59 smithi050 ceph-mon[32753]: pgmap v481: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:01.794 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:01 smithi142 ceph-mon[32944]: pgmap v482: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:01.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:01 smithi046 ceph-mon[29201]: pgmap v482: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:01.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:01 smithi050 ceph-mon[32753]: pgmap v482: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:03.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:03 smithi142 ceph-mon[32944]: pgmap v483: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:03.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:03 smithi046 ceph-mon[29201]: pgmap v483: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:03.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:03 smithi050 ceph-mon[32753]: pgmap v483: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:05.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:05 smithi142 ceph-mon[32944]: pgmap v484: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:05.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:05 smithi046 ceph-mon[29201]: pgmap v484: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:05.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:05 smithi050 ceph-mon[32753]: pgmap v484: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:07.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:07 smithi142 ceph-mon[32944]: pgmap v485: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:07.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:07 smithi046 ceph-mon[29201]: pgmap v485: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:07.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:07 smithi050 ceph-mon[32753]: pgmap v485: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:09.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:09 smithi142 ceph-mon[32944]: pgmap v486: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:09.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:09 smithi046 ceph-mon[29201]: pgmap v486: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:09.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:09 smithi050 ceph-mon[32753]: pgmap v486: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:10.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:10 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:39:10.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:10 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:39:10.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:10 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:39:10.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:10 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:39:10.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:10 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:39:10.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:10 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:39:10.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:10 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:39:10.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:10 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:39:10.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:10 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:39:10.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:10 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:39:10.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:10 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:39:10.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:10 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:39:11.769 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:11 smithi050 ceph-mon[32753]: pgmap v487: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:11.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:11 smithi142 ceph-mon[32944]: pgmap v487: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:11.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:11 smithi046 ceph-mon[29201]: pgmap v487: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:13.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:13 smithi142 ceph-mon[32944]: pgmap v488: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:13.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:13 smithi046 ceph-mon[29201]: pgmap v488: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:13.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:13 smithi050 ceph-mon[32753]: pgmap v488: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:15.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:15 smithi142 ceph-mon[32944]: pgmap v489: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:15.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:15 smithi046 ceph-mon[29201]: pgmap v489: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:15.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:15 smithi050 ceph-mon[32753]: pgmap v489: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:17.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:17 smithi142 ceph-mon[32944]: pgmap v490: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:17.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:17 smithi046 ceph-mon[29201]: pgmap v490: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:17.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:17 smithi050 ceph-mon[32753]: pgmap v490: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:19.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:19 smithi142 ceph-mon[32944]: pgmap v491: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:19.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:19 smithi046 ceph-mon[29201]: pgmap v491: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:19.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:19 smithi050 ceph-mon[32753]: pgmap v491: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:21.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:21 smithi142 ceph-mon[32944]: pgmap v492: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:21.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:21 smithi046 ceph-mon[29201]: pgmap v492: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:21.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:21 smithi050 ceph-mon[32753]: pgmap v492: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:23.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:23 smithi142 ceph-mon[32944]: pgmap v493: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:23.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:23 smithi046 ceph-mon[29201]: pgmap v493: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:23.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:23 smithi050 ceph-mon[32753]: pgmap v493: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:25.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:25 smithi142 ceph-mon[32944]: pgmap v494: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:25.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:25 smithi046 ceph-mon[29201]: pgmap v494: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:25.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:25 smithi050 ceph-mon[32753]: pgmap v494: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:27.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:27 smithi142 ceph-mon[32944]: pgmap v495: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:27.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:27 smithi046 ceph-mon[29201]: pgmap v495: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:27.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:27 smithi050 ceph-mon[32753]: pgmap v495: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:29.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:29 smithi142 ceph-mon[32944]: pgmap v496: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:29.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:29 smithi046 ceph-mon[29201]: pgmap v496: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:29.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:29 smithi050 ceph-mon[32753]: pgmap v496: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:31.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:31 smithi142 ceph-mon[32944]: pgmap v497: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:31.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:31 smithi046 ceph-mon[29201]: pgmap v497: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:31.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:31 smithi050 ceph-mon[32753]: pgmap v497: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:33.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:33 smithi142 ceph-mon[32944]: pgmap v498: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:33.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:33 smithi046 ceph-mon[29201]: pgmap v498: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:33.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:33 smithi050 ceph-mon[32753]: pgmap v498: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:35.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:35 smithi142 ceph-mon[32944]: pgmap v499: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:35.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:35 smithi046 ceph-mon[29201]: pgmap v499: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:35.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:35 smithi050 ceph-mon[32753]: pgmap v499: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:37.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:37 smithi142 ceph-mon[32944]: pgmap v500: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:37.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:37 smithi046 ceph-mon[29201]: pgmap v500: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:37.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:37 smithi050 ceph-mon[32753]: pgmap v500: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:39.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:39 smithi142 ceph-mon[32944]: pgmap v501: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:39.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:39 smithi046 ceph-mon[29201]: pgmap v501: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:39.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:39 smithi050 ceph-mon[32753]: pgmap v501: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:41.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:41 smithi142 ceph-mon[32944]: pgmap v502: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:41.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:41 smithi046 ceph-mon[29201]: pgmap v502: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:41.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:41 smithi050 ceph-mon[32753]: pgmap v502: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:43.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:43 smithi142 ceph-mon[32944]: pgmap v503: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:43.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:43 smithi046 ceph-mon[29201]: pgmap v503: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:43.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:43 smithi050 ceph-mon[32753]: pgmap v503: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:45.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:45 smithi142 ceph-mon[32944]: pgmap v504: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:45.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:45 smithi046 ceph-mon[29201]: pgmap v504: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:45.870 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:45 smithi050 ceph-mon[32753]: pgmap v504: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:47.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:47 smithi142 ceph-mon[32944]: pgmap v505: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:47.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:47 smithi046 ceph-mon[29201]: pgmap v505: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:47.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:47 smithi050 ceph-mon[32753]: pgmap v505: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:49.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:49 smithi142 ceph-mon[32944]: pgmap v506: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:49.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:49 smithi046 ceph-mon[29201]: pgmap v506: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:49.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:49 smithi050 ceph-mon[32753]: pgmap v506: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:51.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:51 smithi142 ceph-mon[32944]: pgmap v507: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:51.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:51 smithi046 ceph-mon[29201]: pgmap v507: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:51.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:51 smithi050 ceph-mon[32753]: pgmap v507: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:53.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:53 smithi142 ceph-mon[32944]: pgmap v508: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:53.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:53 smithi046 ceph-mon[29201]: pgmap v508: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:53.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:53 smithi050 ceph-mon[32753]: pgmap v508: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:55.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:55 smithi142 ceph-mon[32944]: pgmap v509: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:55.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:55 smithi046 ceph-mon[29201]: pgmap v509: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:55.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:55 smithi050 ceph-mon[32753]: pgmap v509: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:57.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:57 smithi142 ceph-mon[32944]: pgmap v510: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:57.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:57 smithi046 ceph-mon[29201]: pgmap v510: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:57.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:57 smithi050 ceph-mon[32753]: pgmap v510: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:39:59.793 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:39:59 smithi142 ceph-mon[32944]: pgmap v511: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:59.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:39:59 smithi046 ceph-mon[29201]: pgmap v511: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:39:59.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:39:59 smithi050 ceph-mon[32753]: pgmap v511: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:00.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:00 smithi142 ceph-mon[32944]: overall HEALTH_OK 2024-04-26T21:40:00.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:00 smithi046 ceph-mon[29201]: overall HEALTH_OK 2024-04-26T21:40:00.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:00 smithi050 ceph-mon[32753]: overall HEALTH_OK 2024-04-26T21:40:01.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:01 smithi142 ceph-mon[32944]: pgmap v512: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:01.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:01 smithi046 ceph-mon[29201]: pgmap v512: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:01.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:01 smithi050 ceph-mon[32753]: pgmap v512: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:03.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:03 smithi142 ceph-mon[32944]: pgmap v513: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:03.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:03 smithi046 ceph-mon[29201]: pgmap v513: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:03.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:03 smithi050 ceph-mon[32753]: pgmap v513: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:05.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:05 smithi142 ceph-mon[32944]: pgmap v514: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:05.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:05 smithi046 ceph-mon[29201]: pgmap v514: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:05.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:05 smithi050 ceph-mon[32753]: pgmap v514: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:07.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:07 smithi142 ceph-mon[32944]: pgmap v515: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:07.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:07 smithi046 ceph-mon[29201]: pgmap v515: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:07.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:07 smithi050 ceph-mon[32753]: pgmap v515: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:09.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:09 smithi142 ceph-mon[32944]: pgmap v516: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:09.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:09 smithi046 ceph-mon[29201]: pgmap v516: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:09.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:09 smithi050 ceph-mon[32753]: pgmap v516: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:10.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:10 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:40:10.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:10 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:40:10.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:10 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:40:11.769 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:11 smithi050 ceph-mon[32753]: pgmap v517: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:11.769 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:11 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:40:11.769 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:11 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:40:11.769 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:11 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:40:11.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:11 smithi142 ceph-mon[32944]: pgmap v517: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:11.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:11 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:40:11.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:11 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:40:11.791 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:11 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:40:11.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:11 smithi046 ceph-mon[29201]: pgmap v517: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:11.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:11 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:40:11.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:11 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:40:11.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:11 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:40:13.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:13 smithi142 ceph-mon[32944]: pgmap v518: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:13.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:13 smithi046 ceph-mon[29201]: pgmap v518: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:13.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:13 smithi050 ceph-mon[32753]: pgmap v518: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:15.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:15 smithi142 ceph-mon[32944]: pgmap v519: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:15.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:15 smithi046 ceph-mon[29201]: pgmap v519: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:15.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:15 smithi050 ceph-mon[32753]: pgmap v519: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:17.793 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:17 smithi142 ceph-mon[32944]: pgmap v520: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:17.804 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:17 smithi046 ceph-mon[29201]: pgmap v520: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:17.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:17 smithi050 ceph-mon[32753]: pgmap v520: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:19.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:19 smithi142 ceph-mon[32944]: pgmap v521: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:19.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:19 smithi046 ceph-mon[29201]: pgmap v521: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:19.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:19 smithi050 ceph-mon[32753]: pgmap v521: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:21.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:21 smithi142 ceph-mon[32944]: pgmap v522: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:21.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:21 smithi046 ceph-mon[29201]: pgmap v522: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:21.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:21 smithi050 ceph-mon[32753]: pgmap v522: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:23.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:23 smithi142 ceph-mon[32944]: pgmap v523: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:23.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:23 smithi046 ceph-mon[29201]: pgmap v523: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:23.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:23 smithi050 ceph-mon[32753]: pgmap v523: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:25.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:25 smithi142 ceph-mon[32944]: pgmap v524: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:25.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:25 smithi046 ceph-mon[29201]: pgmap v524: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:25.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:25 smithi050 ceph-mon[32753]: pgmap v524: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:27.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:27 smithi142 ceph-mon[32944]: pgmap v525: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:27.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:27 smithi046 ceph-mon[29201]: pgmap v525: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:27.867 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:27 smithi050 ceph-mon[32753]: pgmap v525: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:29.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:29 smithi142 ceph-mon[32944]: pgmap v526: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:29.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:29 smithi046 ceph-mon[29201]: pgmap v526: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:29.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:29 smithi050 ceph-mon[32753]: pgmap v526: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:31.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:31 smithi142 ceph-mon[32944]: pgmap v527: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:31.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:31 smithi046 ceph-mon[29201]: pgmap v527: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:31.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:31 smithi050 ceph-mon[32753]: pgmap v527: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:33.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:33 smithi142 ceph-mon[32944]: pgmap v528: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:33.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:33 smithi046 ceph-mon[29201]: pgmap v528: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:33.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:33 smithi050 ceph-mon[32753]: pgmap v528: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:35.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:35 smithi142 ceph-mon[32944]: pgmap v529: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:35.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:35 smithi046 ceph-mon[29201]: pgmap v529: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:35.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:35 smithi050 ceph-mon[32753]: pgmap v529: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:37.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:37 smithi142 ceph-mon[32944]: pgmap v530: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:37.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:37 smithi046 ceph-mon[29201]: pgmap v530: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:37.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:37 smithi050 ceph-mon[32753]: pgmap v530: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:39.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:39 smithi046 ceph-mon[29201]: pgmap v531: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:39.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:39 smithi050 ceph-mon[32753]: pgmap v531: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:40.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:39 smithi142 ceph-mon[32944]: pgmap v531: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:41.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:41 smithi046 ceph-mon[29201]: pgmap v532: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:41.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:41 smithi050 ceph-mon[32753]: pgmap v532: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:42.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:41 smithi142 ceph-mon[32944]: pgmap v532: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:43.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:43 smithi046 ceph-mon[29201]: pgmap v533: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:43.867 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:43 smithi050 ceph-mon[32753]: pgmap v533: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:44.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:43 smithi142 ceph-mon[32944]: pgmap v533: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:45.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:45 smithi046 ceph-mon[29201]: pgmap v534: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:45.843 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:45 smithi142 ceph-mon[32944]: pgmap v534: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:45.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:45 smithi050 ceph-mon[32753]: pgmap v534: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:46.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:40:46.843+0000 7efc617fa640 1 -- 172.21.15.46:0/123926620 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44005850 con 0x7efc64102b30 2024-04-26T21:40:46.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:40:46.844+0000 7efc627fc640 1 -- 172.21.15.46:0/123926620 <== mon.0 v2:172.21.15.46:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc44005850 con 0x7efc64102b30 2024-04-26T21:40:47.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:47 smithi046 ceph-mon[29201]: pgmap v535: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:47.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:47 smithi050 ceph-mon[32753]: pgmap v535: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:48.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:47 smithi142 ceph-mon[32944]: pgmap v535: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:49.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:49 smithi050 ceph-mon[32753]: pgmap v536: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:50.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:49 smithi142 ceph-mon[32944]: pgmap v536: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:50.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:49 smithi046 ceph-mon[29201]: pgmap v536: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:51.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:51 smithi050 ceph-mon[32753]: pgmap v537: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:52.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:51 smithi142 ceph-mon[32944]: pgmap v537: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:52.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:51 smithi046 ceph-mon[29201]: pgmap v537: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:53.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:53 smithi050 ceph-mon[32753]: pgmap v538: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:54.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:53 smithi142 ceph-mon[32944]: pgmap v538: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:54.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:53 smithi046 ceph-mon[29201]: pgmap v538: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:55.856 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:55 smithi046 ceph-mon[29201]: pgmap v539: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:55.867 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:55 smithi050 ceph-mon[32753]: pgmap v539: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:56.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:55 smithi142 ceph-mon[32944]: pgmap v539: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:40:57.867 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:57 smithi050 ceph-mon[32753]: pgmap v540: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:58.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:57 smithi142 ceph-mon[32944]: pgmap v540: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:58.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:57 smithi046 ceph-mon[29201]: pgmap v540: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:40:59.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:40:59 smithi050 ceph-mon[32753]: pgmap v541: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:00.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:40:59 smithi142 ceph-mon[32944]: pgmap v541: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:00.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:40:59 smithi046 ceph-mon[29201]: pgmap v541: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:01.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:01 smithi050 ceph-mon[32753]: pgmap v542: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:02.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:01 smithi142 ceph-mon[32944]: pgmap v542: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:02.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:01 smithi046 ceph-mon[29201]: pgmap v542: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:03.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:03 smithi050 ceph-mon[32753]: pgmap v543: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:04.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:03 smithi142 ceph-mon[32944]: pgmap v543: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:04.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:03 smithi046 ceph-mon[29201]: pgmap v543: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:05.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:05 smithi050 ceph-mon[32753]: pgmap v544: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:06.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:05 smithi142 ceph-mon[32944]: pgmap v544: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:06.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:05 smithi046 ceph-mon[29201]: pgmap v544: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:07.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:07 smithi050 ceph-mon[32753]: pgmap v545: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:08.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:07 smithi142 ceph-mon[32944]: pgmap v545: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:08.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:07 smithi046 ceph-mon[29201]: pgmap v545: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:09.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:09 smithi050 ceph-mon[32753]: pgmap v546: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:10.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:09 smithi142 ceph-mon[32944]: pgmap v546: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:10.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:09 smithi046 ceph-mon[29201]: pgmap v546: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:11.867 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:11 smithi050 ceph-mon[32753]: pgmap v547: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:11.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:11 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:41:12.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:11 smithi142 ceph-mon[32944]: pgmap v547: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:12.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:11 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:41:12.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:11 smithi046 ceph-mon[29201]: pgmap v547: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:12.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:11 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:41:12.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:12 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:41:12.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:12 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:41:12.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:12 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:41:12.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:12 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:41:12.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:12 smithi050 ceph-mon[32753]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-26T21:41:12.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:12 smithi050 ceph-mon[32753]: Unable to set osd_memory_target on smithi046 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-26T21:41:12.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:12 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:41:12.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:12 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:41:12.869 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:12 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:41:13.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:12 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:41:13.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:12 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:41:13.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:12 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:41:13.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:12 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:41:13.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:12 smithi142 ceph-mon[32944]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-26T21:41:13.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:12 smithi142 ceph-mon[32944]: Unable to set osd_memory_target on smithi046 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-26T21:41:13.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:12 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:41:13.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:12 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:41:13.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:12 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:41:13.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:12 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:41:13.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:12 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:41:13.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:12 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:41:13.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:12 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:41:13.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:12 smithi046 ceph-mon[29201]: Adjusting osd_memory_target on smithi046 to 177.2M 2024-04-26T21:41:13.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:12 smithi046 ceph-mon[29201]: Unable to set osd_memory_target on smithi046 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-04-26T21:41:13.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:12 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:41:13.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:12 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:41:13.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:12 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:41:14.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:13 smithi142 ceph-mon[32944]: pgmap v548: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:14.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:13 smithi046 ceph-mon[29201]: pgmap v548: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:14.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:13 smithi050 ceph-mon[32753]: pgmap v548: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:16.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:15 smithi142 ceph-mon[32944]: pgmap v549: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:16.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:15 smithi046 ceph-mon[29201]: pgmap v549: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:16.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:15 smithi050 ceph-mon[32753]: pgmap v549: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:18.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:17 smithi142 ceph-mon[32944]: pgmap v550: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:18.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:17 smithi046 ceph-mon[29201]: pgmap v550: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:18.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:17 smithi050 ceph-mon[32753]: pgmap v550: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:20.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:19 smithi142 ceph-mon[32944]: pgmap v551: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:20.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:19 smithi046 ceph-mon[29201]: pgmap v551: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:20.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:19 smithi050 ceph-mon[32753]: pgmap v551: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:22.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:21 smithi142 ceph-mon[32944]: pgmap v552: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:22.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:21 smithi046 ceph-mon[29201]: pgmap v552: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:22.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:21 smithi050 ceph-mon[32753]: pgmap v552: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:24.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:23 smithi142 ceph-mon[32944]: pgmap v553: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:24.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:23 smithi046 ceph-mon[29201]: pgmap v553: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:24.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:23 smithi050 ceph-mon[32753]: pgmap v553: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:26.293 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:25 smithi142 ceph-mon[32944]: pgmap v554: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:26.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:25 smithi046 ceph-mon[29201]: pgmap v554: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:26.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:25 smithi050 ceph-mon[32753]: pgmap v554: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:28.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:27 smithi142 ceph-mon[32944]: pgmap v555: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:28.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:27 smithi046 ceph-mon[29201]: pgmap v555: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:28.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:27 smithi050 ceph-mon[32753]: pgmap v555: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:30.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:29 smithi142 ceph-mon[32944]: pgmap v556: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:30.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:29 smithi046 ceph-mon[29201]: pgmap v556: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:30.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:29 smithi050 ceph-mon[32753]: pgmap v556: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:32.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:31 smithi142 ceph-mon[32944]: pgmap v557: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:32.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:31 smithi046 ceph-mon[29201]: pgmap v557: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:32.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:31 smithi050 ceph-mon[32753]: pgmap v557: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:34.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:33 smithi142 ceph-mon[32944]: pgmap v558: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:34.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:33 smithi046 ceph-mon[29201]: pgmap v558: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:34.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:33 smithi050 ceph-mon[32753]: pgmap v558: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:36.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:35 smithi142 ceph-mon[32944]: pgmap v559: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:36.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:35 smithi046 ceph-mon[29201]: pgmap v559: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:36.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:35 smithi050 ceph-mon[32753]: pgmap v559: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:38.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:37 smithi142 ceph-mon[32944]: pgmap v560: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:38.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:37 smithi046 ceph-mon[29201]: pgmap v560: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:38.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:37 smithi050 ceph-mon[32753]: pgmap v560: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:40.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:39 smithi142 ceph-mon[32944]: pgmap v561: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:40.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:39 smithi046 ceph-mon[29201]: pgmap v561: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:40.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:39 smithi050 ceph-mon[32753]: pgmap v561: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:42.170 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:41 smithi050 ceph-mon[32753]: pgmap v562: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:42.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:41 smithi142 ceph-mon[32944]: pgmap v562: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:42.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:41 smithi046 ceph-mon[29201]: pgmap v562: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:44.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:43 smithi142 ceph-mon[32944]: pgmap v563: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:44.305 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:43 smithi046 ceph-mon[29201]: pgmap v563: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:44.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:43 smithi050 ceph-mon[32753]: pgmap v563: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:46.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:45 smithi142 ceph-mon[32944]: pgmap v564: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:46.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:45 smithi046 ceph-mon[29201]: pgmap v564: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:46.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:45 smithi050 ceph-mon[32753]: pgmap v564: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:48.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:47 smithi142 ceph-mon[32944]: pgmap v565: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:48.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:47 smithi046 ceph-mon[29201]: pgmap v565: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:48.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:47 smithi050 ceph-mon[32753]: pgmap v565: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:50.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:49 smithi142 ceph-mon[32944]: pgmap v566: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:50.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:49 smithi046 ceph-mon[29201]: pgmap v566: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:50.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:49 smithi050 ceph-mon[32753]: pgmap v566: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:52.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:51 smithi142 ceph-mon[32944]: pgmap v567: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:52.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:51 smithi046 ceph-mon[29201]: pgmap v567: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:52.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:51 smithi050 ceph-mon[32753]: pgmap v567: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:54.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:53 smithi142 ceph-mon[32944]: pgmap v568: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:54.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:53 smithi046 ceph-mon[29201]: pgmap v568: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:54.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:53 smithi050 ceph-mon[32753]: pgmap v568: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:55.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:54 smithi142 ceph-mon[32944]: pgmap v569: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:55.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:54 smithi046 ceph-mon[29201]: pgmap v569: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:55.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:54 smithi050 ceph-mon[32753]: pgmap v569: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:57.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:57 smithi142 ceph-mon[32944]: pgmap v570: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:57.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:57 smithi046 ceph-mon[29201]: pgmap v570: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:57.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:57 smithi050 ceph-mon[32753]: pgmap v570: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:41:59.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:41:59 smithi142 ceph-mon[32944]: pgmap v571: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:59.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:41:59 smithi046 ceph-mon[29201]: pgmap v571: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:41:59.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:41:59 smithi050 ceph-mon[32753]: pgmap v571: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:01.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:01 smithi142 ceph-mon[32944]: pgmap v572: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:01.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:01 smithi046 ceph-mon[29201]: pgmap v572: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:01.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:01 smithi050 ceph-mon[32753]: pgmap v572: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:03.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:03 smithi142 ceph-mon[32944]: pgmap v573: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:03.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:03 smithi046 ceph-mon[29201]: pgmap v573: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:03.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:03 smithi050 ceph-mon[32753]: pgmap v573: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:05.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:05 smithi142 ceph-mon[32944]: pgmap v574: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:05.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:05 smithi046 ceph-mon[29201]: pgmap v574: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:05.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:05 smithi050 ceph-mon[32753]: pgmap v574: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:07.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:07 smithi142 ceph-mon[32944]: pgmap v575: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:07.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:07 smithi046 ceph-mon[29201]: pgmap v575: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:07.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:07 smithi050 ceph-mon[32753]: pgmap v575: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:09.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:09 smithi142 ceph-mon[32944]: pgmap v576: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:09.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:09 smithi046 ceph-mon[29201]: pgmap v576: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:09.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:09 smithi050 ceph-mon[32753]: pgmap v576: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:11.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:11 smithi142 ceph-mon[32944]: pgmap v577: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:11.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:11 smithi046 ceph-mon[29201]: pgmap v577: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:11.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:11 smithi050 ceph-mon[32753]: pgmap v577: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:12.518 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:12 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:42:12.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:12 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:42:12.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:12 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:42:13.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:13 smithi142 ceph-mon[32944]: pgmap v578: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:13.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:13 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:42:13.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:13 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:42:13.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:13 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:42:13.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:13 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:42:13.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:13 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:42:13.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:13 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:42:13.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:13 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:42:13.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:13 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:42:13.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:13 smithi142 ceph-mon[32944]: Adjusting osd_memory_target on smithi050 to 177.2M 2024-04-26T21:42:13.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:13 smithi142 ceph-mon[32944]: Adjusting osd_memory_target on smithi142 to 176.2M 2024-04-26T21:42:13.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:13 smithi142 ceph-mon[32944]: Unable to set osd_memory_target on smithi050 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-26T21:42:13.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:13 smithi142 ceph-mon[32944]: Unable to set osd_memory_target on smithi142 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-26T21:42:13.542 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:13 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:42:13.542 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:13 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:42:13.542 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:13 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:42:13.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:13 smithi046 ceph-mon[29201]: pgmap v578: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:13.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:13 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:42:13.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:13 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:42:13.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:13 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:42:13.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:13 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:42:13.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:13 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:42:13.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:13 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:42:13.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:13 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:42:13.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:13 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:42:13.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:13 smithi046 ceph-mon[29201]: Adjusting osd_memory_target on smithi050 to 177.2M 2024-04-26T21:42:13.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:13 smithi046 ceph-mon[29201]: Adjusting osd_memory_target on smithi142 to 176.2M 2024-04-26T21:42:13.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:13 smithi046 ceph-mon[29201]: Unable to set osd_memory_target on smithi050 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-26T21:42:13.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:13 smithi046 ceph-mon[29201]: Unable to set osd_memory_target on smithi142 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-26T21:42:13.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:13 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:42:13.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:13 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:42:13.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:13 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:42:13.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:13 smithi050 ceph-mon[32753]: pgmap v578: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:13.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:13 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:42:13.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:13 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:42:13.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:13 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:42:13.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:13 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:42:13.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:13 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:42:13.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:13 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:42:13.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:13 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:42:13.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:13 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:42:13.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:13 smithi050 ceph-mon[32753]: Adjusting osd_memory_target on smithi050 to 177.2M 2024-04-26T21:42:13.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:13 smithi050 ceph-mon[32753]: Adjusting osd_memory_target on smithi142 to 176.2M 2024-04-26T21:42:13.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:13 smithi050 ceph-mon[32753]: Unable to set osd_memory_target on smithi050 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-26T21:42:13.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:13 smithi050 ceph-mon[32753]: Unable to set osd_memory_target on smithi142 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-04-26T21:42:13.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:13 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:42:13.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:13 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:42:13.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:13 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:42:15.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:15 smithi142 ceph-mon[32944]: pgmap v579: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:15.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:15 smithi046 ceph-mon[29201]: pgmap v579: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:15.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:15 smithi050 ceph-mon[32753]: pgmap v579: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:17.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:17 smithi142 ceph-mon[32944]: pgmap v580: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:17.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:17 smithi046 ceph-mon[29201]: pgmap v580: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:17.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:17 smithi050 ceph-mon[32753]: pgmap v580: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:19.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:19 smithi142 ceph-mon[32944]: pgmap v581: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:19.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:19 smithi046 ceph-mon[29201]: pgmap v581: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:19.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:19 smithi050 ceph-mon[32753]: pgmap v581: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:21.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:21 smithi142 ceph-mon[32944]: pgmap v582: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:21.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:21 smithi046 ceph-mon[29201]: pgmap v582: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:21.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:21 smithi050 ceph-mon[32753]: pgmap v582: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:23.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:23 smithi142 ceph-mon[32944]: pgmap v583: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:23.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:23 smithi046 ceph-mon[29201]: pgmap v583: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:23.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:23 smithi050 ceph-mon[32753]: pgmap v583: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:25.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:25 smithi142 ceph-mon[32944]: pgmap v584: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:25.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:25 smithi046 ceph-mon[29201]: pgmap v584: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:25.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:25 smithi050 ceph-mon[32753]: pgmap v584: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:27.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:27 smithi142 ceph-mon[32944]: pgmap v585: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:27.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:27 smithi046 ceph-mon[29201]: pgmap v585: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:27.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:27 smithi050 ceph-mon[32753]: pgmap v585: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:29.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:29 smithi142 ceph-mon[32944]: pgmap v586: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:29.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:29 smithi046 ceph-mon[29201]: pgmap v586: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:29.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:29 smithi050 ceph-mon[32753]: pgmap v586: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:31.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:31 smithi142 ceph-mon[32944]: pgmap v587: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:31.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:31 smithi046 ceph-mon[29201]: pgmap v587: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:31.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:31 smithi050 ceph-mon[32753]: pgmap v587: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:33.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:33 smithi142 ceph-mon[32944]: pgmap v588: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:33.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:33 smithi046 ceph-mon[29201]: pgmap v588: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:33.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:33 smithi050 ceph-mon[32753]: pgmap v588: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:35.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:35 smithi142 ceph-mon[32944]: pgmap v589: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:35.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:35 smithi046 ceph-mon[29201]: pgmap v589: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:35.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:35 smithi050 ceph-mon[32753]: pgmap v589: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:37.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:37 smithi142 ceph-mon[32944]: pgmap v590: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:37.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:37 smithi046 ceph-mon[29201]: pgmap v590: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:37.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:37 smithi050 ceph-mon[32753]: pgmap v590: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:39.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:39 smithi142 ceph-mon[32944]: pgmap v591: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:39.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:39 smithi046 ceph-mon[29201]: pgmap v591: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:39.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:39 smithi050 ceph-mon[32753]: pgmap v591: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:41.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:41 smithi142 ceph-mon[32944]: pgmap v592: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:41.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:41 smithi046 ceph-mon[29201]: pgmap v592: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:41.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:41 smithi050 ceph-mon[32753]: pgmap v592: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:43.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:43 smithi142 ceph-mon[32944]: pgmap v593: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:43.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:43 smithi046 ceph-mon[29201]: pgmap v593: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:43.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:43 smithi050 ceph-mon[32753]: pgmap v593: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:45.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:45 smithi142 ceph-mon[32944]: pgmap v594: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:45.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:45 smithi046 ceph-mon[29201]: pgmap v594: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:45.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:45 smithi050 ceph-mon[32753]: pgmap v594: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:47.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:47 smithi142 ceph-mon[32944]: pgmap v595: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:47.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:47 smithi046 ceph-mon[29201]: pgmap v595: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:47.867 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:47 smithi050 ceph-mon[32753]: pgmap v595: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:49.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:49 smithi142 ceph-mon[32944]: pgmap v596: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:49.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:49 smithi046 ceph-mon[29201]: pgmap v596: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:49.867 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:49 smithi050 ceph-mon[32753]: pgmap v596: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:51.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:51 smithi142 ceph-mon[32944]: pgmap v597: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:51.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:51 smithi046 ceph-mon[29201]: pgmap v597: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:51.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:51 smithi050 ceph-mon[32753]: pgmap v597: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:42:53.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:53 smithi142 ceph-mon[32944]: pgmap v598: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:53.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:53 smithi046 ceph-mon[29201]: pgmap v598: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:53.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:53 smithi050 ceph-mon[32753]: pgmap v598: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:55.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:55 smithi142 ceph-mon[32944]: pgmap v599: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:55.817 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:55 smithi046 ceph-mon[29201]: pgmap v599: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:55.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:55 smithi050 ceph-mon[32753]: pgmap v599: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:42:57.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:57 smithi142 ceph-mon[32944]: pgmap v600: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-26T21:42:57.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:57 smithi046 ceph-mon[29201]: pgmap v600: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-26T21:42:57.867 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:57 smithi050 ceph-mon[32753]: pgmap v600: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-26T21:42:59.793 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:42:59 smithi142 ceph-mon[32944]: pgmap v601: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:42:59.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:42:59 smithi046 ceph-mon[29201]: pgmap v601: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:42:59.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:42:59 smithi050 ceph-mon[32753]: pgmap v601: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:43:01.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:01 smithi142 ceph-mon[32944]: pgmap v602: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-26T21:43:01.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:01 smithi046 ceph-mon[29201]: pgmap v602: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-26T21:43:01.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:43:01.863+0000 7efc617fa640 1 -- 172.21.15.46:0/123926620 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7efc44005b70 con 0x7efc64102b30 2024-04-26T21:43:01.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:43:01.863+0000 7efc627fc640 1 -- 172.21.15.46:0/123926620 <== mon.0 v2:172.21.15.46:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7efc44005b70 con 0x7efc64102b30 2024-04-26T21:43:01.867 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:01 smithi050 ceph-mon[32753]: pgmap v602: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-26T21:43:03.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:03 smithi142 ceph-mon[32944]: pgmap v603: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:43:03.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:03 smithi046 ceph-mon[29201]: pgmap v603: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:43:03.867 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:03 smithi050 ceph-mon[32753]: pgmap v603: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:43:05.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:05 smithi142 ceph-mon[32944]: pgmap v604: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:43:05.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:05 smithi046 ceph-mon[29201]: pgmap v604: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:43:05.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:05 smithi050 ceph-mon[32753]: pgmap v604: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:43:07.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:07 smithi142 ceph-mon[32944]: pgmap v605: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-26T21:43:07.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:07 smithi046 ceph-mon[29201]: pgmap v605: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-26T21:43:07.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:07 smithi050 ceph-mon[32753]: pgmap v605: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-04-26T21:43:09.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:09 smithi142 ceph-mon[32944]: pgmap v606: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:09.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:09 smithi046 ceph-mon[29201]: pgmap v606: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:09.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:09 smithi050 ceph-mon[32753]: pgmap v606: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:11.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:11 smithi142 ceph-mon[32944]: pgmap v607: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:11.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:11 smithi046 ceph-mon[29201]: pgmap v607: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:11.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:11 smithi050 ceph-mon[32753]: pgmap v607: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:13.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:13 smithi046 ceph-mon[29201]: pgmap v608: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:13.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:13 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:43:13.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:13 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:43:13.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:13 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:43:13.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:13 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:43:13.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:13 smithi050 ceph-mon[32753]: pgmap v608: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:13.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:13 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:43:13.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:13 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:43:13.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:13 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:43:13.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:13 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:43:14.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:13 smithi142 ceph-mon[32944]: pgmap v608: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:14.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:13 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:43:14.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:13 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:43:14.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:13 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:43:14.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:13 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:43:15.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:15 smithi046 ceph-mon[29201]: pgmap v609: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:15.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:15 smithi050 ceph-mon[32753]: pgmap v609: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:16.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:15 smithi142 ceph-mon[32944]: pgmap v609: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:17.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:17 smithi046 ceph-mon[29201]: pgmap v610: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:17.842 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:17 smithi142 ceph-mon[32944]: pgmap v610: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:17.867 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:17 smithi050 ceph-mon[32753]: pgmap v610: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:19.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:19 smithi046 ceph-mon[29201]: pgmap v611: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:19.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:19 smithi050 ceph-mon[32753]: pgmap v611: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:20.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:19 smithi142 ceph-mon[32944]: pgmap v611: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:21.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:21 smithi046 ceph-mon[29201]: pgmap v612: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:21.867 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:21 smithi050 ceph-mon[32753]: pgmap v612: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:22.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:21 smithi142 ceph-mon[32944]: pgmap v612: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:23.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:23 smithi050 ceph-mon[32753]: pgmap v613: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:24.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:23 smithi142 ceph-mon[32944]: pgmap v613: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:24.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:23 smithi046 ceph-mon[29201]: pgmap v613: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:25.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:25 smithi050 ceph-mon[32753]: pgmap v614: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:26.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:25 smithi142 ceph-mon[32944]: pgmap v614: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:26.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:25 smithi046 ceph-mon[29201]: pgmap v614: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:27.855 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:27 smithi046 ceph-mon[29201]: pgmap v615: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:27.867 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:27 smithi050 ceph-mon[32753]: pgmap v615: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:28.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:27 smithi142 ceph-mon[32944]: pgmap v615: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:29.867 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:29 smithi050 ceph-mon[32753]: pgmap v616: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:30.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:29 smithi142 ceph-mon[32944]: pgmap v616: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:30.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:29 smithi046 ceph-mon[29201]: pgmap v616: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:31.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:31 smithi050 ceph-mon[32753]: pgmap v617: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:32.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:31 smithi142 ceph-mon[32944]: pgmap v617: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:32.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:31 smithi046 ceph-mon[29201]: pgmap v617: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:33.867 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:33 smithi050 ceph-mon[32753]: pgmap v618: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:34.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:33 smithi142 ceph-mon[32944]: pgmap v618: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:34.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:33 smithi046 ceph-mon[29201]: pgmap v618: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:35.867 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:35 smithi050 ceph-mon[32753]: pgmap v619: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:36.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:35 smithi142 ceph-mon[32944]: pgmap v619: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:36.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:35 smithi046 ceph-mon[29201]: pgmap v619: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:37.867 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:37 smithi050 ceph-mon[32753]: pgmap v620: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:38.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:37 smithi142 ceph-mon[32944]: pgmap v620: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:38.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:37 smithi046 ceph-mon[29201]: pgmap v620: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:39.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:39 smithi050 ceph-mon[32753]: pgmap v621: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:40.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:39 smithi142 ceph-mon[32944]: pgmap v621: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:40.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:39 smithi046 ceph-mon[29201]: pgmap v621: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:41.867 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:41 smithi050 ceph-mon[32753]: pgmap v622: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:42.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:41 smithi142 ceph-mon[32944]: pgmap v622: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:42.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:41 smithi046 ceph-mon[29201]: pgmap v622: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:43.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:43 smithi050 ceph-mon[32753]: pgmap v623: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:44.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:43 smithi142 ceph-mon[32944]: pgmap v623: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:44.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:43 smithi046 ceph-mon[29201]: pgmap v623: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:45.867 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:45 smithi050 ceph-mon[32753]: pgmap v624: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:46.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:45 smithi142 ceph-mon[32944]: pgmap v624: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:46.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:45 smithi046 ceph-mon[29201]: pgmap v624: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:46.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:43:46.869+0000 7efc617fa640 1 -- 172.21.15.46:0/123926620 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44005ed0 con 0x7efc64102b30 2024-04-26T21:43:46.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:43:46.870+0000 7efc627fc640 1 -- 172.21.15.46:0/123926620 <== mon.0 v2:172.21.15.46:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc44005ed0 con 0x7efc64102b30 2024-04-26T21:43:47.843 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:47 smithi142 ceph-mon[32944]: pgmap v625: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:47.867 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:47 smithi050 ceph-mon[32753]: pgmap v625: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:48.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:47 smithi046 ceph-mon[29201]: pgmap v625: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:49.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:49 smithi050 ceph-mon[32753]: pgmap v626: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:50.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:49 smithi142 ceph-mon[32944]: pgmap v626: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:50.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:49 smithi046 ceph-mon[29201]: pgmap v626: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:51.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:51 smithi050 ceph-mon[32753]: pgmap v627: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:52.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:51 smithi142 ceph-mon[32944]: pgmap v627: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:52.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:51 smithi046 ceph-mon[29201]: pgmap v627: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:53.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:53 smithi050 ceph-mon[32753]: pgmap v628: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:54.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:53 smithi142 ceph-mon[32944]: pgmap v628: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:54.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:53 smithi046 ceph-mon[29201]: pgmap v628: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:55.867 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:55 smithi050 ceph-mon[32753]: pgmap v629: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:56.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:55 smithi142 ceph-mon[32944]: pgmap v629: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:56.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:55 smithi046 ceph-mon[29201]: pgmap v629: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:43:57.856 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:57 smithi046 ceph-mon[29201]: pgmap v630: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:57.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:57 smithi050 ceph-mon[32753]: pgmap v630: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:58.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:57 smithi142 ceph-mon[32944]: pgmap v630: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:43:59.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:43:59 smithi050 ceph-mon[32753]: pgmap v631: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:00.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:43:59 smithi142 ceph-mon[32944]: pgmap v631: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:00.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:43:59 smithi046 ceph-mon[29201]: pgmap v631: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:01.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:01 smithi050 ceph-mon[32753]: pgmap v632: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:02.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:01 smithi142 ceph-mon[32944]: pgmap v632: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:02.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:01 smithi046 ceph-mon[29201]: pgmap v632: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:03.867 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:03 smithi050 ceph-mon[32753]: pgmap v633: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:04.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:03 smithi142 ceph-mon[32944]: pgmap v633: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:04.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:03 smithi046 ceph-mon[29201]: pgmap v633: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:06.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:05 smithi142 ceph-mon[32944]: pgmap v634: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:06.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:05 smithi046 ceph-mon[29201]: pgmap v634: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:06.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:05 smithi050 ceph-mon[32753]: pgmap v634: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:08.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:07 smithi142 ceph-mon[32944]: pgmap v635: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:08.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:07 smithi046 ceph-mon[29201]: pgmap v635: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:08.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:07 smithi050 ceph-mon[32753]: pgmap v635: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:10.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:09 smithi142 ceph-mon[32944]: pgmap v636: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:10.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:09 smithi046 ceph-mon[29201]: pgmap v636: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:10.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:09 smithi050 ceph-mon[32753]: pgmap v636: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:12.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:11 smithi142 ceph-mon[32944]: pgmap v637: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:12.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:11 smithi046 ceph-mon[29201]: pgmap v637: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:12.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:11 smithi050 ceph-mon[32753]: pgmap v637: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:13.920 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:13 smithi050 ceph-mon[32753]: pgmap v638: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:14.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:13 smithi142 ceph-mon[32944]: pgmap v638: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:14.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:13 smithi046 ceph-mon[29201]: pgmap v638: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:15.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:14 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:44:15.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:14 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:44:15.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:14 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:44:15.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:14 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:44:15.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:14 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:44:15.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:14 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:44:15.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:14 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:44:15.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:14 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:44:15.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:14 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:44:15.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:14 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:44:15.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:14 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:44:15.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:14 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:44:16.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:15 smithi142 ceph-mon[32944]: pgmap v639: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:44:16.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:15 smithi046 ceph-mon[29201]: pgmap v639: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:44:16.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:15 smithi050 ceph-mon[32753]: pgmap v639: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:44:18.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:17 smithi142 ceph-mon[32944]: pgmap v640: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:18.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:17 smithi046 ceph-mon[29201]: pgmap v640: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:18.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:17 smithi050 ceph-mon[32753]: pgmap v640: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:20.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:19 smithi142 ceph-mon[32944]: pgmap v641: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:20.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:19 smithi046 ceph-mon[29201]: pgmap v641: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:20.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:19 smithi050 ceph-mon[32753]: pgmap v641: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:22.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:21 smithi142 ceph-mon[32944]: pgmap v642: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:22.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:21 smithi046 ceph-mon[29201]: pgmap v642: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:22.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:21 smithi050 ceph-mon[32753]: pgmap v642: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:24.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:23 smithi142 ceph-mon[32944]: pgmap v643: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:24.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:23 smithi046 ceph-mon[29201]: pgmap v643: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:24.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:23 smithi050 ceph-mon[32753]: pgmap v643: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:26.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:25 smithi142 ceph-mon[32944]: pgmap v644: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:44:26.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:25 smithi046 ceph-mon[29201]: pgmap v644: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:44:26.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:25 smithi050 ceph-mon[32753]: pgmap v644: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:44:28.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:27 smithi142 ceph-mon[32944]: pgmap v645: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:28.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:27 smithi046 ceph-mon[29201]: pgmap v645: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:28.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:27 smithi050 ceph-mon[32753]: pgmap v645: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:30.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:29 smithi142 ceph-mon[32944]: pgmap v646: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:30.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:29 smithi046 ceph-mon[29201]: pgmap v646: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:30.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:29 smithi050 ceph-mon[32753]: pgmap v646: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:32.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:31 smithi142 ceph-mon[32944]: pgmap v647: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:32.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:31 smithi046 ceph-mon[29201]: pgmap v647: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:32.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:31 smithi050 ceph-mon[32753]: pgmap v647: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:34.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:33 smithi142 ceph-mon[32944]: pgmap v648: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:34.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:33 smithi046 ceph-mon[29201]: pgmap v648: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:34.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:33 smithi050 ceph-mon[32753]: pgmap v648: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:36.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:35 smithi142 ceph-mon[32944]: pgmap v649: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:44:36.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:35 smithi046 ceph-mon[29201]: pgmap v649: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:44:36.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:35 smithi050 ceph-mon[32753]: pgmap v649: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:44:38.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:37 smithi142 ceph-mon[32944]: pgmap v650: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:38.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:37 smithi046 ceph-mon[29201]: pgmap v650: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:38.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:37 smithi050 ceph-mon[32753]: pgmap v650: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:40.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:39 smithi142 ceph-mon[32944]: pgmap v651: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:40.052 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:39 smithi046 ceph-mon[29201]: pgmap v651: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:40.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:39 smithi050 ceph-mon[32753]: pgmap v651: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:42.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:41 smithi142 ceph-mon[32944]: pgmap v652: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:42.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:41 smithi046 ceph-mon[29201]: pgmap v652: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:42.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:41 smithi050 ceph-mon[32753]: pgmap v652: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:44.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:43 smithi142 ceph-mon[32944]: pgmap v653: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:44.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:43 smithi046 ceph-mon[29201]: pgmap v653: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:44.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:43 smithi050 ceph-mon[32753]: pgmap v653: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:46.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:45 smithi142 ceph-mon[32944]: pgmap v654: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:44:46.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:45 smithi046 ceph-mon[29201]: pgmap v654: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:44:46.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:45 smithi050 ceph-mon[32753]: pgmap v654: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:44:48.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:47 smithi142 ceph-mon[32944]: pgmap v655: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:48.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:47 smithi046 ceph-mon[29201]: pgmap v655: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:48.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:47 smithi050 ceph-mon[32753]: pgmap v655: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:50.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:49 smithi142 ceph-mon[32944]: pgmap v656: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:50.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:49 smithi046 ceph-mon[29201]: pgmap v656: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:50.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:49 smithi050 ceph-mon[32753]: pgmap v656: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:52.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:51 smithi142 ceph-mon[32944]: pgmap v657: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:52.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:51 smithi046 ceph-mon[29201]: pgmap v657: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:52.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:51 smithi050 ceph-mon[32753]: pgmap v657: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:54.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:53 smithi142 ceph-mon[32944]: pgmap v658: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:54.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:53 smithi046 ceph-mon[29201]: pgmap v658: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:54.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:53 smithi050 ceph-mon[32753]: pgmap v658: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:44:56.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:55 smithi142 ceph-mon[32944]: pgmap v659: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:44:56.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:55 smithi046 ceph-mon[29201]: pgmap v659: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:44:56.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:55 smithi050 ceph-mon[32753]: pgmap v659: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:44:58.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:57 smithi142 ceph-mon[32944]: pgmap v660: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:58.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:57 smithi046 ceph-mon[29201]: pgmap v660: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:44:58.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:57 smithi050 ceph-mon[32753]: pgmap v660: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:00.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:44:59 smithi142 ceph-mon[32944]: pgmap v661: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:00.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:44:59 smithi046 ceph-mon[29201]: pgmap v661: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:00.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:44:59 smithi050 ceph-mon[32753]: pgmap v661: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:02.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:01 smithi142 ceph-mon[32944]: pgmap v662: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:02.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:01 smithi046 ceph-mon[29201]: pgmap v662: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:02.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:01 smithi050 ceph-mon[32753]: pgmap v662: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:04.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:03 smithi142 ceph-mon[32944]: pgmap v663: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:04.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:03 smithi046 ceph-mon[29201]: pgmap v663: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:04.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:03 smithi050 ceph-mon[32753]: pgmap v663: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:06.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:05 smithi142 ceph-mon[32944]: pgmap v664: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:45:06.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:05 smithi046 ceph-mon[29201]: pgmap v664: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:45:06.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:05 smithi050 ceph-mon[32753]: pgmap v664: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:45:08.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:07 smithi142 ceph-mon[32944]: pgmap v665: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:08.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:07 smithi046 ceph-mon[29201]: pgmap v665: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:08.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:07 smithi050 ceph-mon[32753]: pgmap v665: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:10.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:09 smithi142 ceph-mon[32944]: pgmap v666: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:10.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:09 smithi046 ceph-mon[29201]: pgmap v666: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:10.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:09 smithi050 ceph-mon[32753]: pgmap v666: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:12.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:11 smithi142 ceph-mon[32944]: pgmap v667: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:12.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:11 smithi046 ceph-mon[29201]: pgmap v667: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:12.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:11 smithi050 ceph-mon[32753]: pgmap v667: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:14.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:13 smithi142 ceph-mon[32944]: pgmap v668: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:14.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:13 smithi046 ceph-mon[29201]: pgmap v668: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:14.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:13 smithi050 ceph-mon[32753]: pgmap v668: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:15.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:14 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:45:15.052 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:14 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:45:15.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:14 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:45:16.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:15 smithi142 ceph-mon[32944]: pgmap v669: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:45:16.052 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:15 smithi046 ceph-mon[29201]: pgmap v669: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:45:16.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:15 smithi050 ceph-mon[32753]: pgmap v669: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:45:17.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:17 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:45:17.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:17 smithi142 ceph-mon[32944]: pgmap v670: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:45:17.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:17 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:45:17.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:17 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:45:17.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:17 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:45:17.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:17 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:45:17.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:17 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:45:17.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:17 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:45:17.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:17 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:45:17.541 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:17 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:45:17.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:17 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:45:17.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:17 smithi046 ceph-mon[29201]: pgmap v670: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:45:17.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:17 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:45:17.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:17 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:45:17.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:17 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:45:17.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:17 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:45:17.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:17 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:45:17.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:17 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:45:17.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:17 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:45:17.554 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:17 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:45:17.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:17 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:45:17.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:17 smithi050 ceph-mon[32753]: pgmap v670: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:45:17.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:17 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:45:17.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:17 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:45:17.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:17 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:45:17.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:17 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:45:17.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:17 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:45:17.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:17 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:45:17.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:17 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:45:17.619 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:17 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:45:19.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:19 smithi142 ceph-mon[32944]: pgmap v671: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:19.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:19 smithi046 ceph-mon[29201]: pgmap v671: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:19.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:19 smithi050 ceph-mon[32753]: pgmap v671: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:21.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:21 smithi142 ceph-mon[32944]: pgmap v672: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:21.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:21 smithi046 ceph-mon[29201]: pgmap v672: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:21.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:21 smithi050 ceph-mon[32753]: pgmap v672: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:23.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:23 smithi142 ceph-mon[32944]: pgmap v673: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:23.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:23 smithi046 ceph-mon[29201]: pgmap v673: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:23.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:23 smithi050 ceph-mon[32753]: pgmap v673: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:25.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:25 smithi142 ceph-mon[32944]: pgmap v674: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:45:25.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:25 smithi046 ceph-mon[29201]: pgmap v674: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:45:25.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:25 smithi050 ceph-mon[32753]: pgmap v674: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:45:27.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:27 smithi142 ceph-mon[32944]: pgmap v675: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:27.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:27 smithi046 ceph-mon[29201]: pgmap v675: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:27.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:27 smithi050 ceph-mon[32753]: pgmap v675: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:29.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:29 smithi142 ceph-mon[32944]: pgmap v676: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:29.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:29 smithi046 ceph-mon[29201]: pgmap v676: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:29.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:29 smithi050 ceph-mon[32753]: pgmap v676: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:31.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:31 smithi142 ceph-mon[32944]: pgmap v677: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:31.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:31 smithi046 ceph-mon[29201]: pgmap v677: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:31.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:31 smithi050 ceph-mon[32753]: pgmap v677: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:33.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:33 smithi142 ceph-mon[32944]: pgmap v678: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:33.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:33 smithi046 ceph-mon[29201]: pgmap v678: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:33.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:33 smithi050 ceph-mon[32753]: pgmap v678: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:35.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:35 smithi142 ceph-mon[32944]: pgmap v679: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:45:35.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:35 smithi046 ceph-mon[29201]: pgmap v679: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:45:35.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:35 smithi050 ceph-mon[32753]: pgmap v679: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:45:37.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:37 smithi142 ceph-mon[32944]: pgmap v680: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:37.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:37 smithi046 ceph-mon[29201]: pgmap v680: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:37.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:37 smithi050 ceph-mon[32753]: pgmap v680: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:39.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:39 smithi142 ceph-mon[32944]: pgmap v681: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:39.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:39 smithi046 ceph-mon[29201]: pgmap v681: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:39.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:39 smithi050 ceph-mon[32753]: pgmap v681: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:41.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:41 smithi142 ceph-mon[32944]: pgmap v682: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:41.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:41 smithi046 ceph-mon[29201]: pgmap v682: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:41.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:41 smithi050 ceph-mon[32753]: pgmap v682: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:43.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:43 smithi142 ceph-mon[32944]: pgmap v683: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:43.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:43 smithi046 ceph-mon[29201]: pgmap v683: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:43.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:43 smithi050 ceph-mon[32753]: pgmap v683: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:45.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:45 smithi142 ceph-mon[32944]: pgmap v684: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:45:45.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:45 smithi046 ceph-mon[29201]: pgmap v684: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:45:45.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:45 smithi050 ceph-mon[32753]: pgmap v684: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:45:47.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:47 smithi142 ceph-mon[32944]: pgmap v685: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:47.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:47 smithi046 ceph-mon[29201]: pgmap v685: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:47.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:47 smithi050 ceph-mon[32753]: pgmap v685: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:49.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:49 smithi142 ceph-mon[32944]: pgmap v686: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:49.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:49 smithi046 ceph-mon[29201]: pgmap v686: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:49.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:49 smithi050 ceph-mon[32753]: pgmap v686: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:51.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:51 smithi142 ceph-mon[32944]: pgmap v687: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:51.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:51 smithi046 ceph-mon[29201]: pgmap v687: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:51.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:51 smithi050 ceph-mon[32753]: pgmap v687: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:53.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:53 smithi142 ceph-mon[32944]: pgmap v688: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:53.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:53 smithi046 ceph-mon[29201]: pgmap v688: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:53.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:53 smithi050 ceph-mon[32753]: pgmap v688: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:55.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:55 smithi142 ceph-mon[32944]: pgmap v689: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:45:55.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:55 smithi046 ceph-mon[29201]: pgmap v689: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:45:55.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:55 smithi050 ceph-mon[32753]: pgmap v689: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:45:57.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:57 smithi142 ceph-mon[32944]: pgmap v690: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:57.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:57 smithi046 ceph-mon[29201]: pgmap v690: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:57.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:57 smithi050 ceph-mon[32753]: pgmap v690: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:45:59.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:45:59 smithi142 ceph-mon[32944]: pgmap v691: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:59.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:45:59 smithi046 ceph-mon[29201]: pgmap v691: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:45:59.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:45:59 smithi050 ceph-mon[32753]: pgmap v691: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:01.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:01 smithi142 ceph-mon[32944]: pgmap v692: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:01.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:01 smithi046 ceph-mon[29201]: pgmap v692: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:01.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:01 smithi050 ceph-mon[32753]: pgmap v692: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:03.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:03 smithi142 ceph-mon[32944]: pgmap v693: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:03.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:03 smithi046 ceph-mon[29201]: pgmap v693: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:03.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:03 smithi050 ceph-mon[32753]: pgmap v693: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:05.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:05 smithi142 ceph-mon[32944]: pgmap v694: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:46:05.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:05 smithi046 ceph-mon[29201]: pgmap v694: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:46:05.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:05 smithi050 ceph-mon[32753]: pgmap v694: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:46:07.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:07 smithi142 ceph-mon[32944]: pgmap v695: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:07.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:07 smithi046 ceph-mon[29201]: pgmap v695: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:07.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:07 smithi050 ceph-mon[32753]: pgmap v695: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:09.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:09 smithi142 ceph-mon[32944]: pgmap v696: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:09.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:09 smithi046 ceph-mon[29201]: pgmap v696: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:09.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:09 smithi050 ceph-mon[32753]: pgmap v696: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:11.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:11 smithi142 ceph-mon[32944]: pgmap v697: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:11.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:11 smithi046 ceph-mon[29201]: pgmap v697: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:11.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:11 smithi050 ceph-mon[32753]: pgmap v697: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:13.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:13 smithi142 ceph-mon[32944]: pgmap v698: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:13.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:13 smithi046 ceph-mon[29201]: pgmap v698: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:13.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:13 smithi050 ceph-mon[32753]: pgmap v698: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:15.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:15 smithi142 ceph-mon[32944]: pgmap v699: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:46:15.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:15 smithi046 ceph-mon[29201]: pgmap v699: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:46:15.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:15 smithi050 ceph-mon[32753]: pgmap v699: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:46:17.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:17 smithi046 ceph-mon[29201]: pgmap v700: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:17.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:17 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:46:17.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:17 smithi050 ceph-mon[32753]: pgmap v700: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:17.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:17 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:46:17.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:17 smithi142 ceph-mon[32944]: pgmap v700: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:17.790 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:17 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:46:19.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:18 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:46:19.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:18 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:46:19.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:18 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:46:19.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:18 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:46:19.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:18 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:46:19.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:18 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:46:19.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:18 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:46:19.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:18 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:46:19.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:18 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:46:19.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:18 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:46:19.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:18 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:46:19.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:18 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:46:19.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:18 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:46:19.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:18 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:46:19.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:18 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:46:19.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:18 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:46:19.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:18 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:46:19.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:18 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:46:19.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:18 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:46:19.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:18 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:46:19.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:18 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:46:19.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:18 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:46:19.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:18 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:46:19.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:18 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:46:19.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:18 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:46:19.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:18 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:46:19.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:18 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:46:20.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:19 smithi050 ceph-mon[32753]: pgmap v701: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:20.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:19 smithi142 ceph-mon[32944]: pgmap v701: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:20.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:19 smithi046 ceph-mon[29201]: pgmap v701: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:22.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:21 smithi050 ceph-mon[32753]: pgmap v702: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:22.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:21 smithi142 ceph-mon[32944]: pgmap v702: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:22.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:21 smithi046 ceph-mon[29201]: pgmap v702: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:24.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:23 smithi050 ceph-mon[32753]: pgmap v703: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:24.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:23 smithi142 ceph-mon[32944]: pgmap v703: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:24.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:23 smithi046 ceph-mon[29201]: pgmap v703: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:26.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:25 smithi050 ceph-mon[32753]: pgmap v704: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:46:26.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:25 smithi142 ceph-mon[32944]: pgmap v704: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:46:26.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:25 smithi046 ceph-mon[29201]: pgmap v704: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:46:28.105 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:27 smithi046 ceph-mon[29201]: pgmap v705: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:28.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:27 smithi050 ceph-mon[32753]: pgmap v705: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:28.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:27 smithi142 ceph-mon[32944]: pgmap v705: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:30.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:29 smithi050 ceph-mon[32753]: pgmap v706: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:30.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:29 smithi142 ceph-mon[32944]: pgmap v706: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:30.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:29 smithi046 ceph-mon[29201]: pgmap v706: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:32.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:31 smithi050 ceph-mon[32753]: pgmap v707: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:32.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:31 smithi142 ceph-mon[32944]: pgmap v707: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:32.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:31 smithi046 ceph-mon[29201]: pgmap v707: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:34.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:33 smithi050 ceph-mon[32753]: pgmap v708: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:34.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:33 smithi142 ceph-mon[32944]: pgmap v708: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:34.302 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:33 smithi046 ceph-mon[29201]: pgmap v708: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:36.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:35 smithi050 ceph-mon[32753]: pgmap v709: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:46:36.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:35 smithi142 ceph-mon[32944]: pgmap v709: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:46:36.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:35 smithi046 ceph-mon[29201]: pgmap v709: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:46:38.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:37 smithi050 ceph-mon[32753]: pgmap v710: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:38.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:37 smithi142 ceph-mon[32944]: pgmap v710: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:38.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:37 smithi046 ceph-mon[29201]: pgmap v710: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:40.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:39 smithi050 ceph-mon[32753]: pgmap v711: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:40.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:39 smithi142 ceph-mon[32944]: pgmap v711: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:40.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:39 smithi046 ceph-mon[29201]: pgmap v711: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:42.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:41 smithi050 ceph-mon[32753]: pgmap v712: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:42.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:41 smithi142 ceph-mon[32944]: pgmap v712: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:42.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:41 smithi046 ceph-mon[29201]: pgmap v712: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:44.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:43 smithi050 ceph-mon[32753]: pgmap v713: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:44.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:43 smithi142 ceph-mon[32944]: pgmap v713: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:44.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:43 smithi046 ceph-mon[29201]: pgmap v713: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:46.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:45 smithi050 ceph-mon[32753]: pgmap v714: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:46:46.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:45 smithi142 ceph-mon[32944]: pgmap v714: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:46:46.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:45 smithi046 ceph-mon[29201]: pgmap v714: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:46:46.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:46:46.896+0000 7efc617fa640 1 -- 172.21.15.46:0/123926620 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc440060f0 con 0x7efc64102b30 2024-04-26T21:46:46.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:46:46.896+0000 7efc627fc640 1 -- 172.21.15.46:0/123926620 <== mon.0 v2:172.21.15.46:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc440060f0 con 0x7efc64102b30 2024-04-26T21:46:48.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:47 smithi050 ceph-mon[32753]: pgmap v715: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:48.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:47 smithi142 ceph-mon[32944]: pgmap v715: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:48.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:47 smithi046 ceph-mon[29201]: pgmap v715: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:50.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:49 smithi050 ceph-mon[32753]: pgmap v716: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:50.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:49 smithi142 ceph-mon[32944]: pgmap v716: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:50.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:49 smithi046 ceph-mon[29201]: pgmap v716: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:52.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:51 smithi050 ceph-mon[32753]: pgmap v717: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:52.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:51 smithi142 ceph-mon[32944]: pgmap v717: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:52.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:51 smithi046 ceph-mon[29201]: pgmap v717: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:54.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:53 smithi142 ceph-mon[32944]: pgmap v718: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:54.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:53 smithi046 ceph-mon[29201]: pgmap v718: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:54.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:53 smithi050 ceph-mon[32753]: pgmap v718: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:46:56.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:55 smithi142 ceph-mon[32944]: pgmap v719: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:46:56.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:55 smithi046 ceph-mon[29201]: pgmap v719: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:46:56.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:55 smithi050 ceph-mon[32753]: pgmap v719: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:46:58.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:57 smithi142 ceph-mon[32944]: pgmap v720: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:58.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:57 smithi046 ceph-mon[29201]: pgmap v720: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:46:58.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:57 smithi050 ceph-mon[32753]: pgmap v720: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:00.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:46:59 smithi142 ceph-mon[32944]: pgmap v721: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:00.302 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:46:59 smithi046 ceph-mon[29201]: pgmap v721: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:00.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:46:59 smithi050 ceph-mon[32753]: pgmap v721: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:02.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:01 smithi142 ceph-mon[32944]: pgmap v722: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:02.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:01 smithi046 ceph-mon[29201]: pgmap v722: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:02.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:01 smithi050 ceph-mon[32753]: pgmap v722: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:04.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:03 smithi142 ceph-mon[32944]: pgmap v723: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:04.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:03 smithi046 ceph-mon[29201]: pgmap v723: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:04.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:03 smithi050 ceph-mon[32753]: pgmap v723: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:06.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:05 smithi142 ceph-mon[32944]: pgmap v724: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:47:06.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:05 smithi046 ceph-mon[29201]: pgmap v724: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:47:06.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:05 smithi050 ceph-mon[32753]: pgmap v724: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:47:08.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:07 smithi142 ceph-mon[32944]: pgmap v725: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:08.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:07 smithi046 ceph-mon[29201]: pgmap v725: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:08.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:07 smithi050 ceph-mon[32753]: pgmap v725: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:10.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:09 smithi142 ceph-mon[32944]: pgmap v726: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:10.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:09 smithi046 ceph-mon[29201]: pgmap v726: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:10.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:09 smithi050 ceph-mon[32753]: pgmap v726: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:12.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:11 smithi142 ceph-mon[32944]: pgmap v727: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:12.302 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:11 smithi046 ceph-mon[29201]: pgmap v727: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:12.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:11 smithi050 ceph-mon[32753]: pgmap v727: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:14.170 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:13 smithi050 ceph-mon[32753]: pgmap v728: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:14.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:13 smithi142 ceph-mon[32944]: pgmap v728: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:14.302 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:13 smithi046 ceph-mon[29201]: pgmap v728: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:16.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:15 smithi142 ceph-mon[32944]: pgmap v729: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:47:16.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:15 smithi046 ceph-mon[29201]: pgmap v729: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:47:16.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:15 smithi050 ceph-mon[32753]: pgmap v729: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:47:18.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:17 smithi142 ceph-mon[32944]: pgmap v730: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:47:18.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:17 smithi046 ceph-mon[29201]: pgmap v730: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:47:18.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:17 smithi050 ceph-mon[32753]: pgmap v730: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:47:19.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:18 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:47:19.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:18 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:47:19.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:18 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:47:20.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:19 smithi142 ceph-mon[32944]: pgmap v731: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:47:20.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:19 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:47:20.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:19 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:47:20.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:19 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:47:20.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:19 smithi046 ceph-mon[29201]: pgmap v731: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:47:20.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:19 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:47:20.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:19 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:47:20.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:19 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:47:20.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:19 smithi050 ceph-mon[32753]: pgmap v731: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:47:20.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:19 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:47:20.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:19 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:47:20.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:19 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:47:22.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:21 smithi142 ceph-mon[32944]: pgmap v732: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:22.302 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:21 smithi046 ceph-mon[29201]: pgmap v732: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:22.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:21 smithi050 ceph-mon[32753]: pgmap v732: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:24.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:23 smithi142 ceph-mon[32944]: pgmap v733: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:47:24.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:23 smithi046 ceph-mon[29201]: pgmap v733: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:47:24.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:23 smithi050 ceph-mon[32753]: pgmap v733: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-04-26T21:47:26.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:25 smithi142 ceph-mon[32944]: pgmap v734: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-26T21:47:26.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:25 smithi046 ceph-mon[29201]: pgmap v734: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-26T21:47:26.369 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:25 smithi050 ceph-mon[32753]: pgmap v734: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-26T21:47:28.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:27 smithi142 ceph-mon[32944]: pgmap v735: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:47:28.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:27 smithi046 ceph-mon[29201]: pgmap v735: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:47:28.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:27 smithi050 ceph-mon[32753]: pgmap v735: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:47:30.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:29 smithi142 ceph-mon[32944]: pgmap v736: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:30.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:29 smithi046 ceph-mon[29201]: pgmap v736: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:30.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:29 smithi050 ceph-mon[32753]: pgmap v736: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:32.294 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:31 smithi142 ceph-mon[32944]: pgmap v737: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:32.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:31 smithi046 ceph-mon[29201]: pgmap v737: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:32.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:31 smithi050 ceph-mon[32753]: pgmap v737: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:34.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:33 smithi142 ceph-mon[32944]: pgmap v738: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:34.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:33 smithi046 ceph-mon[29201]: pgmap v738: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:34.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:33 smithi050 ceph-mon[32753]: pgmap v738: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:36.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:35 smithi142 ceph-mon[32944]: pgmap v739: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:47:36.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:35 smithi046 ceph-mon[29201]: pgmap v739: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:47:36.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:35 smithi050 ceph-mon[32753]: pgmap v739: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:47:38.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:37 smithi142 ceph-mon[32944]: pgmap v740: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:38.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:37 smithi046 ceph-mon[29201]: pgmap v740: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:38.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:37 smithi050 ceph-mon[32753]: pgmap v740: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:40.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:39 smithi142 ceph-mon[32944]: pgmap v741: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:40.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:39 smithi046 ceph-mon[29201]: pgmap v741: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:40.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:39 smithi050 ceph-mon[32753]: pgmap v741: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:42.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:41 smithi142 ceph-mon[32944]: pgmap v742: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:42.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:41 smithi046 ceph-mon[29201]: pgmap v742: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:42.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:41 smithi050 ceph-mon[32753]: pgmap v742: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:44.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:43 smithi142 ceph-mon[32944]: pgmap v743: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:44.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:43 smithi046 ceph-mon[29201]: pgmap v743: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:44.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:43 smithi050 ceph-mon[32753]: pgmap v743: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:46.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:45 smithi142 ceph-mon[32944]: pgmap v744: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:47:46.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:45 smithi046 ceph-mon[29201]: pgmap v744: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:47:46.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:45 smithi050 ceph-mon[32753]: pgmap v744: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:47:48.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:47 smithi142 ceph-mon[32944]: pgmap v745: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:48.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:47 smithi046 ceph-mon[29201]: pgmap v745: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:48.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:47 smithi050 ceph-mon[32753]: pgmap v745: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:50.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:49 smithi142 ceph-mon[32944]: pgmap v746: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:50.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:49 smithi046 ceph-mon[29201]: pgmap v746: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:50.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:49 smithi050 ceph-mon[32753]: pgmap v746: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:52.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:51 smithi142 ceph-mon[32944]: pgmap v747: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:52.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:51 smithi046 ceph-mon[29201]: pgmap v747: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:52.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:51 smithi050 ceph-mon[32753]: pgmap v747: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:54.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:53 smithi142 ceph-mon[32944]: pgmap v748: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:54.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:53 smithi046 ceph-mon[29201]: pgmap v748: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:54.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:53 smithi050 ceph-mon[32753]: pgmap v748: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:47:56.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:55 smithi142 ceph-mon[32944]: pgmap v749: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:47:56.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:55 smithi046 ceph-mon[29201]: pgmap v749: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:47:56.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:55 smithi050 ceph-mon[32753]: pgmap v749: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:47:58.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:57 smithi142 ceph-mon[32944]: pgmap v750: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:58.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:57 smithi046 ceph-mon[29201]: pgmap v750: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:47:58.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:57 smithi050 ceph-mon[32753]: pgmap v750: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:00.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:47:59 smithi142 ceph-mon[32944]: pgmap v751: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:00.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:47:59 smithi046 ceph-mon[29201]: pgmap v751: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:00.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:47:59 smithi050 ceph-mon[32753]: pgmap v751: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:02.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:01 smithi142 ceph-mon[32944]: pgmap v752: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:02.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:01 smithi046 ceph-mon[29201]: pgmap v752: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:02.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:01 smithi050 ceph-mon[32753]: pgmap v752: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:04.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:03 smithi142 ceph-mon[32944]: pgmap v753: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:04.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:03 smithi046 ceph-mon[29201]: pgmap v753: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:04.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:03 smithi050 ceph-mon[32753]: pgmap v753: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:06.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:05 smithi142 ceph-mon[32944]: pgmap v754: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:48:06.302 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:05 smithi046 ceph-mon[29201]: pgmap v754: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:48:06.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:05 smithi050 ceph-mon[32753]: pgmap v754: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:48:08.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:07 smithi142 ceph-mon[32944]: pgmap v755: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:08.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:07 smithi046 ceph-mon[29201]: pgmap v755: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:08.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:07 smithi050 ceph-mon[32753]: pgmap v755: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:10.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:09 smithi142 ceph-mon[32944]: pgmap v756: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:10.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:09 smithi046 ceph-mon[29201]: pgmap v756: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:10.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:09 smithi050 ceph-mon[32753]: pgmap v756: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:12.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:11 smithi142 ceph-mon[32944]: pgmap v757: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:12.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:11 smithi046 ceph-mon[29201]: pgmap v757: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:12.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:11 smithi050 ceph-mon[32753]: pgmap v757: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:14.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:13 smithi142 ceph-mon[32944]: pgmap v758: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:14.302 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:13 smithi046 ceph-mon[29201]: pgmap v758: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:14.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:13 smithi050 ceph-mon[32753]: pgmap v758: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:16.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:15 smithi142 ceph-mon[32944]: pgmap v759: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:48:16.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:15 smithi046 ceph-mon[29201]: pgmap v759: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:48:16.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:15 smithi050 ceph-mon[32753]: pgmap v759: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:48:18.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:17 smithi142 ceph-mon[32944]: pgmap v760: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:18.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:17 smithi046 ceph-mon[29201]: pgmap v760: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:18.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:17 smithi050 ceph-mon[32753]: pgmap v760: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:20.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:20 smithi142 ceph-mon[32944]: pgmap v761: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:20.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:20 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:48:20.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:20 smithi046 ceph-mon[29201]: pgmap v761: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:20.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:20 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:48:20.367 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:20 smithi050 ceph-mon[32753]: pgmap v761: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:20.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:20 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:48:21.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:21 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:48:21.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:21 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:48:21.290 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:21 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:48:21.291 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:21 smithi142 ceph-mon[32944]: pgmap v762: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:21.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:21 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:48:21.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:21 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:48:21.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:21 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:48:21.303 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:21 smithi046 ceph-mon[29201]: pgmap v762: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:21.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:21 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:48:21.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:21 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:48:21.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:21 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:48:21.368 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:21 smithi050 ceph-mon[32753]: pgmap v762: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:23.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:23 smithi142 ceph-mon[32944]: pgmap v763: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:23.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:23 smithi046 ceph-mon[29201]: pgmap v763: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:23.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:23 smithi050 ceph-mon[32753]: pgmap v763: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:25.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:25 smithi142 ceph-mon[32944]: pgmap v764: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:48:25.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:25 smithi046 ceph-mon[29201]: pgmap v764: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:48:25.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:25 smithi050 ceph-mon[32753]: pgmap v764: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:48:27.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:27 smithi142 ceph-mon[32944]: pgmap v765: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:27.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:27 smithi046 ceph-mon[29201]: pgmap v765: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:27.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:27 smithi050 ceph-mon[32753]: pgmap v765: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:29.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:29 smithi142 ceph-mon[32944]: pgmap v766: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:29.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:29 smithi046 ceph-mon[29201]: pgmap v766: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:29.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:29 smithi050 ceph-mon[32753]: pgmap v766: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:31.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:31 smithi142 ceph-mon[32944]: pgmap v767: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:31.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:31 smithi046 ceph-mon[29201]: pgmap v767: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:31.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:31 smithi050 ceph-mon[32753]: pgmap v767: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:33.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:33 smithi142 ceph-mon[32944]: pgmap v768: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:33.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:33 smithi046 ceph-mon[29201]: pgmap v768: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:33.618 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:33 smithi050 ceph-mon[32753]: pgmap v768: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:35.540 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:35 smithi142 ceph-mon[32944]: pgmap v769: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:48:35.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:35 smithi046 ceph-mon[29201]: pgmap v769: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:48:35.617 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:35 smithi050 ceph-mon[32753]: pgmap v769: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:48:37.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:37 smithi050 ceph-mon[32753]: pgmap v770: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:38.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:37 smithi142 ceph-mon[32944]: pgmap v770: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:38.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:37 smithi046 ceph-mon[29201]: pgmap v770: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:39.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:39 smithi050 ceph-mon[32753]: pgmap v771: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:40.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:39 smithi142 ceph-mon[32944]: pgmap v771: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:40.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:39 smithi046 ceph-mon[29201]: pgmap v771: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:41.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:41 smithi050 ceph-mon[32753]: pgmap v772: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:42.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:41 smithi142 ceph-mon[32944]: pgmap v772: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:42.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:41 smithi046 ceph-mon[29201]: pgmap v772: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:43.867 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:43 smithi050 ceph-mon[32753]: pgmap v773: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:44.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:43 smithi142 ceph-mon[32944]: pgmap v773: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:44.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:43 smithi046 ceph-mon[29201]: pgmap v773: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:45.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:45 smithi050 ceph-mon[32753]: pgmap v774: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:48:46.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:45 smithi142 ceph-mon[32944]: pgmap v774: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:48:46.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:45 smithi046 ceph-mon[29201]: pgmap v774: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:48:47.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:47 smithi050 ceph-mon[32753]: pgmap v775: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:48.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:47 smithi142 ceph-mon[32944]: pgmap v775: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:48.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:47 smithi046 ceph-mon[29201]: pgmap v775: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:49.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:49 smithi050 ceph-mon[32753]: pgmap v776: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:50.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:49 smithi142 ceph-mon[32944]: pgmap v776: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:50.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:49 smithi046 ceph-mon[29201]: pgmap v776: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:51.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:51 smithi050 ceph-mon[32753]: pgmap v777: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:52.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:51 smithi142 ceph-mon[32944]: pgmap v777: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:52.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:51 smithi046 ceph-mon[29201]: pgmap v777: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:53.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:53 smithi050 ceph-mon[32753]: pgmap v778: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:54.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:53 smithi142 ceph-mon[32944]: pgmap v778: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:54.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:53 smithi046 ceph-mon[29201]: pgmap v778: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:48:55.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:55 smithi050 ceph-mon[32753]: pgmap v779: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:48:56.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:55 smithi142 ceph-mon[32944]: pgmap v779: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:48:56.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:55 smithi046 ceph-mon[29201]: pgmap v779: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:48:57.867 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:57 smithi050 ceph-mon[32753]: pgmap v780: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:58.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:57 smithi142 ceph-mon[32944]: pgmap v780: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:58.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:57 smithi046 ceph-mon[29201]: pgmap v780: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:48:59.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:48:59 smithi050 ceph-mon[32753]: pgmap v781: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:00.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:48:59 smithi142 ceph-mon[32944]: pgmap v781: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:00.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:48:59 smithi046 ceph-mon[29201]: pgmap v781: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:01.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:01 smithi050 ceph-mon[32753]: pgmap v782: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:49:02.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:01 smithi142 ceph-mon[32944]: pgmap v782: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:49:02.052 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:01 smithi046 ceph-mon[29201]: pgmap v782: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:49:03.868 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:03 smithi050 ceph-mon[32753]: pgmap v783: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:04.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:03 smithi142 ceph-mon[32944]: pgmap v783: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:04.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:03 smithi046 ceph-mon[29201]: pgmap v783: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:06.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:05 smithi142 ceph-mon[32944]: pgmap v784: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:49:06.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:05 smithi046 ceph-mon[29201]: pgmap v784: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:49:06.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:05 smithi050 ceph-mon[32753]: pgmap v784: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:49:08.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:07 smithi142 ceph-mon[32944]: pgmap v785: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:49:08.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:07 smithi046 ceph-mon[29201]: pgmap v785: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:49:08.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:07 smithi050 ceph-mon[32753]: pgmap v785: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:49:10.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:09 smithi142 ceph-mon[32944]: pgmap v786: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:10.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:09 smithi046 ceph-mon[29201]: pgmap v786: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:10.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:09 smithi050 ceph-mon[32753]: pgmap v786: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:12.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:11 smithi142 ceph-mon[32944]: pgmap v787: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:49:12.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:11 smithi046 ceph-mon[29201]: pgmap v787: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:49:12.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:11 smithi050 ceph-mon[32753]: pgmap v787: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:49:14.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:13 smithi142 ceph-mon[32944]: pgmap v788: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:14.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:13 smithi046 ceph-mon[29201]: pgmap v788: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:14.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:13 smithi050 ceph-mon[32753]: pgmap v788: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:15.920 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:15 smithi050 ceph-mon[32753]: pgmap v789: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:49:16.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:15 smithi142 ceph-mon[32944]: pgmap v789: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:49:16.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:15 smithi046 ceph-mon[29201]: pgmap v789: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:49:18.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:17 smithi142 ceph-mon[32944]: pgmap v790: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:49:18.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:17 smithi046 ceph-mon[29201]: pgmap v790: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:49:18.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:17 smithi050 ceph-mon[32753]: pgmap v790: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:49:20.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:19 smithi142 ceph-mon[32944]: pgmap v791: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:20.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:19 smithi046 ceph-mon[29201]: pgmap v791: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:20.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:19 smithi050 ceph-mon[32753]: pgmap v791: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:21.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:20 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:49:21.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:20 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:49:21.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:20 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:49:22.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:21 smithi142 ceph-mon[32944]: pgmap v792: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:49:22.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:21 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:49:22.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:21 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:49:22.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:21 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:49:22.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:21 smithi046 ceph-mon[29201]: pgmap v792: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:49:22.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:21 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:49:22.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:21 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:49:22.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:21 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:49:22.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:21 smithi050 ceph-mon[32753]: pgmap v792: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:49:22.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:21 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:49:22.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:21 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:49:22.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:21 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' 2024-04-26T21:49:24.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:23 smithi142 ceph-mon[32944]: pgmap v793: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:24.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:23 smithi046 ceph-mon[29201]: pgmap v793: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:24.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:23 smithi050 ceph-mon[32753]: pgmap v793: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:26.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:25 smithi142 ceph-mon[32944]: pgmap v794: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:49:26.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:25 smithi046 ceph-mon[29201]: pgmap v794: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:49:26.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:25 smithi050 ceph-mon[32753]: pgmap v794: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:49:28.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:27 smithi142 ceph-mon[32944]: pgmap v795: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:49:28.052 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:27 smithi046 ceph-mon[29201]: pgmap v795: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:49:28.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:27 smithi050 ceph-mon[32753]: pgmap v795: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:49:30.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:29 smithi142 ceph-mon[32944]: pgmap v796: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:30.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:29 smithi046 ceph-mon[29201]: pgmap v796: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:30.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:29 smithi050 ceph-mon[32753]: pgmap v796: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:32.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:31 smithi142 ceph-mon[32944]: pgmap v797: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:49:32.052 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:31 smithi046 ceph-mon[29201]: pgmap v797: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:49:32.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:31 smithi050 ceph-mon[32753]: pgmap v797: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:49:34.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:33 smithi142 ceph-mon[32944]: pgmap v798: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:34.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:33 smithi046 ceph-mon[29201]: pgmap v798: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:34.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:33 smithi050 ceph-mon[32753]: pgmap v798: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:36.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:35 smithi142 ceph-mon[32944]: pgmap v799: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:49:36.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:35 smithi046 ceph-mon[29201]: pgmap v799: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:49:36.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:35 smithi050 ceph-mon[32753]: pgmap v799: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:49:38.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:37 smithi142 ceph-mon[32944]: pgmap v800: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:49:38.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:37 smithi046 ceph-mon[29201]: pgmap v800: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:49:38.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:37 smithi050 ceph-mon[32753]: pgmap v800: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:49:40.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:39 smithi142 ceph-mon[32944]: pgmap v801: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:40.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:39 smithi046 ceph-mon[29201]: pgmap v801: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:40.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:39 smithi050 ceph-mon[32753]: pgmap v801: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:42.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:41 smithi142 ceph-mon[32944]: pgmap v802: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:49:42.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:41 smithi046 ceph-mon[29201]: pgmap v802: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:49:42.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:41 smithi050 ceph-mon[32753]: pgmap v802: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:49:44.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:43 smithi142 ceph-mon[32944]: pgmap v803: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:44.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:43 smithi046 ceph-mon[29201]: pgmap v803: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:44.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:43 smithi050 ceph-mon[32753]: pgmap v803: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:46.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:45 smithi142 ceph-mon[32944]: pgmap v804: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:49:46.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:45 smithi046 ceph-mon[29201]: pgmap v804: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:49:46.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:45 smithi050 ceph-mon[32753]: pgmap v804: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:49:46.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:46.823+0000 7efc695e3640 1 -- 172.21.15.46:0/123926620 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7efc4c0a3cf0 msgr2=0x7efc4c0a61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900027811) for more than 900000000 us, fault. 2024-04-26T21:49:46.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:46.823+0000 7efc695e3640 1 --2- 172.21.15.46:0/123926620 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7efc4c0a3cf0 0x7efc4c0a61b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7efc64102990 tx=0x7efc5c005f90 comp rx=0 tx=0).stop 2024-04-26T21:49:46.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:46.823+0000 7efc627fc640 1 -- 172.21.15.46:0/123926620 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7efc4c0a3cf0 msgr2=0x7efc4c0a61b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T21:49:46.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:46.823+0000 7efc627fc640 1 --2- 172.21.15.46:0/123926620 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7efc4c0a3cf0 0x7efc4c0a61b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:46.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:46.823+0000 7efc627fc640 1 --2- 172.21.15.46:0/123926620 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7efc4c0adc80 0x7efc4c0b0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:46.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:46.823+0000 7efc627fc640 1 -- 172.21.15.46:0/123926620 --> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7efc58012ec0 con 0x7efc4c0adc80 2024-04-26T21:49:46.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:46.823+0000 7efc695e3640 1 --2- 172.21.15.46:0/123926620 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7efc4c0adc80 0x7efc4c0b0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:46.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:46.824+0000 7efc695e3640 1 --2- 172.21.15.46:0/123926620 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7efc4c0adc80 0x7efc4c0b0070 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7efc5c005fd0 tx=0x7efc5c017000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:46.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:46.921+0000 7efc617fa640 1 -- 172.21.15.46:0/123926620 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7efc44006360 con 0x7efc64102b30 2024-04-26T21:49:46.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:46.921+0000 7efc627fc640 1 -- 172.21.15.46:0/123926620 <== mon.0 v2:172.21.15.46:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7efc44006360 con 0x7efc64102b30 2024-04-26T21:49:47.015 INFO:teuthology.orchestra.run.smithi046.stderr:Error ETIMEDOUT: Connection timed out. Please check the client's permission and connection. 2024-04-26T21:49:47.021 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-04-26T21:49:47.022 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi046 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10'" 2024-04-26T21:49:47.248 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=a76c6789f7584bd494aa1f704fd701ab Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi046 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10'" 2024-04-26T21:49:47.250 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-26T21:49:47.262 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-26T21:49:47.263 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-26T21:49:47.263 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T21:49:47.286 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-04-26T21:49:47.286 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T21:49:47.345 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:49:47.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.501+0000 7f079ca17640 1 -- 172.21.15.46:0/2105848550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0798102a80 msgr2=0x7f0798102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:47.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.501+0000 7f079ca17640 1 --2- 172.21.15.46:0/2105848550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0798102a80 0x7f0798102f00 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0780009900 tx=0x7f078002a410 comp rx=0 tx=0).stop 2024-04-26T21:49:47.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.501+0000 7f079ca17640 1 -- 172.21.15.46:0/2105848550 shutdown_connections 2024-04-26T21:49:47.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.501+0000 7f079ca17640 1 --2- 172.21.15.46:0/2105848550 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0798103440 0x7f0798109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:47.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.501+0000 7f079ca17640 1 --2- 172.21.15.46:0/2105848550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0798102a80 0x7f0798102f00 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:47.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.501+0000 7f079ca17640 1 --2- 172.21.15.46:0/2105848550 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0798101890 0x7f0798101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:47.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.501+0000 7f079ca17640 1 -- 172.21.15.46:0/2105848550 >> 172.21.15.46:0/2105848550 conn(0x7f07980fce20 msgr2=0x7f07980ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:47.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.501+0000 7f079ca17640 1 -- 172.21.15.46:0/2105848550 shutdown_connections 2024-04-26T21:49:47.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.501+0000 7f079ca17640 1 -- 172.21.15.46:0/2105848550 wait complete. 2024-04-26T21:49:47.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.502+0000 7f079ca17640 1 Processor -- start 2024-04-26T21:49:47.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.502+0000 7f079ca17640 1 -- start start 2024-04-26T21:49:47.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.502+0000 7f079ca17640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0798101890 0x7f07981057d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:47.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.502+0000 7f079ca17640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0798102a80 0x7f0798103e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:47.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.502+0000 7f079ca17640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0798103440 0x7f07981043a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:47.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.502+0000 7f079ca17640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f079810d3a0 con 0x7f0798102a80 2024-04-26T21:49:47.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.502+0000 7f079ca17640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f079810d220 con 0x7f0798101890 2024-04-26T21:49:47.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.502+0000 7f079ca17640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f079810d520 con 0x7f0798103440 2024-04-26T21:49:47.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.502+0000 7f078ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0798102a80 0x7f0798103e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:47.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.502+0000 7f078ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0798102a80 0x7f0798103e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44718/0 (socket says 172.21.15.46:44718) 2024-04-26T21:49:47.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.502+0000 7f078ffff640 1 -- 172.21.15.46:0/3399598926 learned_addr learned my addr 172.21.15.46:0/3399598926 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:49:47.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.502+0000 7f079659c640 1 --2- 172.21.15.46:0/3399598926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0798101890 0x7f07981057d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:47.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.502+0000 7f0796d9d640 1 --2- 172.21.15.46:0/3399598926 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0798103440 0x7f07981043a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:47.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.503+0000 7f078ffff640 1 -- 172.21.15.46:0/3399598926 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0798103440 msgr2=0x7f07981043a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:47.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.503+0000 7f078ffff640 1 --2- 172.21.15.46:0/3399598926 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0798103440 0x7f07981043a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:47.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.503+0000 7f078ffff640 1 -- 172.21.15.46:0/3399598926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0798101890 msgr2=0x7f07981057d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:47.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.503+0000 7f078ffff640 1 --2- 172.21.15.46:0/3399598926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0798101890 0x7f07981057d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:47.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.503+0000 7f078ffff640 1 -- 172.21.15.46:0/3399598926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0798104c60 con 0x7f0798102a80 2024-04-26T21:49:47.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.503+0000 7f078ffff640 1 --2- 172.21.15.46:0/3399598926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0798102a80 0x7f0798103e60 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0780005940 tx=0x7f0780004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:47.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.503+0000 7f078f7fe640 1 -- 172.21.15.46:0/3399598926 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078003f030 con 0x7f0798102a80 2024-04-26T21:49:47.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.503+0000 7f079ca17640 1 -- 172.21.15.46:0/3399598926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07981a2ce0 con 0x7f0798102a80 2024-04-26T21:49:47.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.503+0000 7f078f7fe640 1 -- 172.21.15.46:0/3399598926 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0780003cb0 con 0x7f0798102a80 2024-04-26T21:49:47.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.503+0000 7f078f7fe640 1 -- 172.21.15.46:0/3399598926 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0780035070 con 0x7f0798102a80 2024-04-26T21:49:47.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.503+0000 7f079ca17640 1 -- 172.21.15.46:0/3399598926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07981a31a0 con 0x7f0798102a80 2024-04-26T21:49:47.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.504+0000 7f079ca17640 1 -- 172.21.15.46:0/3399598926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07981a3430 con 0x7f0798102a80 2024-04-26T21:49:47.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.505+0000 7f078f7fe640 1 -- 172.21.15.46:0/3399598926 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0780042030 con 0x7f0798102a80 2024-04-26T21:49:47.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.506+0000 7f078f7fe640 1 --2- 172.21.15.46:0/3399598926 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f07840a3c40 0x7f07840a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:47.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.506+0000 7f078f7fe640 1 -- 172.21.15.46:0/3399598926 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f07800f0dc0 con 0x7f0798102a80 2024-04-26T21:49:47.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.506+0000 7f079659c640 1 --2- 172.21.15.46:0/3399598926 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f07840a3c40 0x7f07840a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:47.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.507+0000 7f079659c640 1 --2- 172.21.15.46:0/3399598926 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f07840a3c40 0x7f07840a6100 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f07981028e0 tx=0x7f078800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:47.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.511+0000 7f078f7fe640 1 -- 172.21.15.46:0/3399598926 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0780003e60 con 0x7f0798102a80 2024-04-26T21:49:47.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.699+0000 7f079ca17640 1 -- 172.21.15.46:0/3399598926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f07981a3b50 con 0x7f0798102a80 2024-04-26T21:49:47.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.700+0000 7f078f7fe640 1 -- 172.21.15.46:0/3399598926 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22602 (secure 0 0 0) 0x7f07800bd020 con 0x7f0798102a80 2024-04-26T21:49:47.701 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:49:47.701 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":87,"fsid":"36c7013a-0413-11ef-bc93-c7b262605968","created":"2024-04-26T21:24:30.481156+0000","modified":"2024-04-26T21:33:31.051554+0000","last_up_change":"2024-04-26T21:31:50.100491+0000","last_in_change":"2024-04-26T21:31:29.643428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:27:57.935988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:32:42.195211+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:32:43.162114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f53c8702-e4da-435f-a9dc-3911d0d36ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6803","nonce":2762665125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6805","nonce":2762665125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6809","nonce":2762665125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6807","nonce":2762665125}]},"public_addr":"172.21.15.46:6803/2762665125","cluster_addr":"172.21.15.46:6805/2762665125","heartbeat_back_addr":"172.21.15.46:6809/2762665125","heartbeat_front_addr":"172.21.15.46:6807/2762665125","state":["exists","up"]},{"osd":1,"uuid":"fc4d6f52-fc8a-4efd-80c9-c092b57e7f42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6801","nonce":1628008970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6803","nonce":1628008970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6807","nonce":1628008970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6805","nonce":1628008970}]},"public_addr":"172.21.15.50:6801/1628008970","cluster_addr":"172.21.15.50:6803/1628008970","heartbeat_back_addr":"172.21.15.50:6807/1628008970","heartbeat_front_addr":"172.21.15.50:6805/1628008970","state":["exists","up"]},{"osd":2,"uuid":"26d1baa1-4ea1-4b7e-9fcf-4f8bbf7329c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6801","nonce":1513279137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6803","nonce":1513279137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6807","nonce":1513279137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6805","nonce":1513279137}]},"public_addr":"172.21.15.142:6801/1513279137","cluster_addr":"172.21.15.142:6803/1513279137","heartbeat_back_addr":"172.21.15.142:6807/1513279137","heartbeat_front_addr":"172.21.15.142:6805/1513279137","state":["exists","up"]},{"osd":3,"uuid":"34d2401f-5fcc-491c-8fcf-c3c492611f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6811","nonce":2435181760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6813","nonce":2435181760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6817","nonce":2435181760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6815","nonce":2435181760}]},"public_addr":"172.21.15.46:6811/2435181760","cluster_addr":"172.21.15.46:6813/2435181760","heartbeat_back_addr":"172.21.15.46:6817/2435181760","heartbeat_front_addr":"172.21.15.46:6815/2435181760","state":["exists","up"]},{"osd":4,"uuid":"986bb709-e4cf-400d-982e-c3f5ac413e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6809","nonce":3177333250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6811","nonce":3177333250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6815","nonce":3177333250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6813","nonce":3177333250}]},"public_addr":"172.21.15.50:6809/3177333250","cluster_addr":"172.21.15.50:6811/3177333250","heartbeat_back_addr":"172.21.15.50:6815/3177333250","heartbeat_front_addr":"172.21.15.50:6813/3177333250","state":["exists","up"]},{"osd":5,"uuid":"b7d65851-ebe1-45e2-833d-8022446f2698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6809","nonce":4113107737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6811","nonce":4113107737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6815","nonce":4113107737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6813","nonce":4113107737}]},"public_addr":"172.21.15.142:6809/4113107737","cluster_addr":"172.21.15.142:6811/4113107737","heartbeat_back_addr":"172.21.15.142:6815/4113107737","heartbeat_front_addr":"172.21.15.142:6813/4113107737","state":["exists","up"]},{"osd":6,"uuid":"2a3e96d7-d5bf-44ff-9134-29f1420c0e4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6819","nonce":2325162882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6821","nonce":2325162882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6825","nonce":2325162882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6823","nonce":2325162882}]},"public_addr":"172.21.15.46:6819/2325162882","cluster_addr":"172.21.15.46:6821/2325162882","heartbeat_back_addr":"172.21.15.46:6825/2325162882","heartbeat_front_addr":"172.21.15.46:6823/2325162882","state":["exists","up"]},{"osd":7,"uuid":"e53ae885-23b6-46fd-b4b1-c62f31a6921f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6817","nonce":820310228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6819","nonce":820310228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6823","nonce":820310228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6821","nonce":820310228}]},"public_addr":"172.21.15.50:6817/820310228","cluster_addr":"172.21.15.50:6819/820310228","heartbeat_back_addr":"172.21.15.50:6823/820310228","heartbeat_front_addr":"172.21.15.50:6821/820310228","state":["exists","up"]},{"osd":8,"uuid":"356e2692-16a9-4045-b58f-c8f9cb5d7241","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6817","nonce":1824918711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6819","nonce":1824918711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6823","nonce":1824918711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6821","nonce":1824918711}]},"public_addr":"172.21.15.142:6817/1824918711","cluster_addr":"172.21.15.142:6819/1824918711","heartbeat_back_addr":"172.21.15.142:6823/1824918711","heartbeat_front_addr":"172.21.15.142:6821/1824918711","state":["exists","up"]},{"osd":9,"uuid":"8e55570e-77e5-4892-8b9f-77dcd03a4eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6827","nonce":3601947736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6829","nonce":3601947736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6833","nonce":3601947736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6831","nonce":3601947736}]},"public_addr":"172.21.15.46:6827/3601947736","cluster_addr":"172.21.15.46:6829/3601947736","heartbeat_back_addr":"172.21.15.46:6833/3601947736","heartbeat_front_addr":"172.21.15.46:6831/3601947736","state":["exists","up"]},{"osd":10,"uuid":"366389fa-fb9a-408b-b5c2-343c054c5a64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6825","nonce":3135765006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6827","nonce":3135765006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6831","nonce":3135765006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6829","nonce":3135765006}]},"public_addr":"172.21.15.50:6825/3135765006","cluster_addr":"172.21.15.50:6827/3135765006","heartbeat_back_addr":"172.21.15.50:6831/3135765006","heartbeat_front_addr":"172.21.15.50:6829/3135765006","state":["exists","up"]},{"osd":11,"uuid":"7b6e3d93-59e1-4853-b4a2-261b728c47a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6825","nonce":1482712612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6827","nonce":1482712612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6831","nonce":1482712612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6829","nonce":1482712612}]},"public_addr":"172.21.15.142:6825/1482712612","cluster_addr":"172.21.15.142:6827/1482712612","heartbeat_back_addr":"172.21.15.142:6831/1482712612","heartbeat_front_addr":"172.21.15.142:6829/1482712612","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:08.082015+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:32.634928+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:55.978864+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:21.796863+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:46.862518+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:11.440432+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:37.764777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:03.374958+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:28.714052+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:55.761167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:22.330951+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:47.839334+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/2465242088":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6801/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6800/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/1381997090":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/3507268217":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2570024356":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/2691061938":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2775954801":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6800/2896098005":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6801/2896098005":"2024-04-27T21:25:02.078730+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:49:47.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.702+0000 7f079ca17640 1 -- 172.21.15.46:0/3399598926 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f07840a3c40 msgr2=0x7f07840a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:47.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.702+0000 7f079ca17640 1 --2- 172.21.15.46:0/3399598926 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f07840a3c40 0x7f07840a6100 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f07981028e0 tx=0x7f078800b390 comp rx=0 tx=0).stop 2024-04-26T21:49:47.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.702+0000 7f079ca17640 1 -- 172.21.15.46:0/3399598926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0798102a80 msgr2=0x7f0798103e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:47.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.702+0000 7f079ca17640 1 --2- 172.21.15.46:0/3399598926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0798102a80 0x7f0798103e60 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0780005940 tx=0x7f0780004430 comp rx=0 tx=0).stop 2024-04-26T21:49:47.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.702+0000 7f079ca17640 1 -- 172.21.15.46:0/3399598926 shutdown_connections 2024-04-26T21:49:47.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.702+0000 7f079ca17640 1 --2- 172.21.15.46:0/3399598926 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f07840a3c40 0x7f07840a6100 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:47.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.702+0000 7f079ca17640 1 --2- 172.21.15.46:0/3399598926 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0798103440 0x7f07981043a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:47.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.702+0000 7f079ca17640 1 --2- 172.21.15.46:0/3399598926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0798102a80 0x7f0798103e60 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:47.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.702+0000 7f079ca17640 1 --2- 172.21.15.46:0/3399598926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0798101890 0x7f07981057d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:47.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.702+0000 7f079ca17640 1 -- 172.21.15.46:0/3399598926 >> 172.21.15.46:0/3399598926 conn(0x7f07980fce20 msgr2=0x7f07980fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:47.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.702+0000 7f079ca17640 1 -- 172.21.15.46:0/3399598926 shutdown_connections 2024-04-26T21:49:47.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.702+0000 7f079ca17640 1 -- 172.21.15.46:0/3399598926 wait complete. 2024-04-26T21:49:47.714 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T21:27:57.935988+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T21:32:42.195211+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T21:32:43.162114+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T21:49:47.715 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T21:49:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.820+0000 7f08bfbb4640 1 -- 172.21.15.46:0/3772838710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08b8108670 msgr2=0x7f08b810aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.820+0000 7f08bfbb4640 1 --2- 172.21.15.46:0/3772838710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08b8108670 0x7f08b810aac0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f08b0009ca0 tx=0x7f08b002a8c0 comp rx=0 tx=0).stop 2024-04-26T21:49:47.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.820+0000 7f08bfbb4640 1 -- 172.21.15.46:0/3772838710 shutdown_connections 2024-04-26T21:49:47.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.820+0000 7f08bfbb4640 1 --2- 172.21.15.46:0/3772838710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08b8108670 0x7f08b810aac0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:47.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.820+0000 7f08bfbb4640 1 --2- 172.21.15.46:0/3772838710 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f08b8105d40 0x7f08b8108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:47.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.820+0000 7f08bfbb4640 1 --2- 172.21.15.46:0/3772838710 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08b8103410 0x7f08b8105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:47.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.820+0000 7f08bfbb4640 1 -- 172.21.15.46:0/3772838710 >> 172.21.15.46:0/3772838710 conn(0x7f08b80fce20 msgr2=0x7f08b80ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:47.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.820+0000 7f08bfbb4640 1 -- 172.21.15.46:0/3772838710 shutdown_connections 2024-04-26T21:49:47.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.820+0000 7f08bfbb4640 1 -- 172.21.15.46:0/3772838710 wait complete. 2024-04-26T21:49:47.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.820+0000 7f08bfbb4640 1 Processor -- start 2024-04-26T21:49:47.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.820+0000 7f08bfbb4640 1 -- start start 2024-04-26T21:49:47.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.821+0000 7f08bfbb4640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f08b8103410 0x7f08b8102d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:47.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.821+0000 7f08bfbb4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08b8105d40 0x7f08b81013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:47.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.821+0000 7f08bfbb4640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08b8108670 0x7f08b8101900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:47.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.821+0000 7f08bfbb4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f08b810e530 con 0x7f08b8105d40 2024-04-26T21:49:47.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.821+0000 7f08bfbb4640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f08b810e3b0 con 0x7f08b8108670 2024-04-26T21:49:47.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.821+0000 7f08bfbb4640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f08b810e6b0 con 0x7f08b8103410 2024-04-26T21:49:47.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.821+0000 7f08bd14f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08b8105d40 0x7f08b81013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:47.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.821+0000 7f08bd14f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08b8105d40 0x7f08b81013c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44736/0 (socket says 172.21.15.46:44736) 2024-04-26T21:49:47.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.821+0000 7f08bd14f640 1 -- 172.21.15.46:0/4278173264 learned_addr learned my addr 172.21.15.46:0/4278173264 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:49:47.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.821+0000 7f08bd950640 1 --2- 172.21.15.46:0/4278173264 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f08b8103410 0x7f08b8102d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:47.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.821+0000 7f08bd14f640 1 -- 172.21.15.46:0/4278173264 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f08b8103410 msgr2=0x7f08b8102d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:47.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.821+0000 7f08be151640 1 --2- 172.21.15.46:0/4278173264 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08b8108670 0x7f08b8101900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:47.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.821+0000 7f08bd14f640 1 --2- 172.21.15.46:0/4278173264 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f08b8103410 0x7f08b8102d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:47.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.821+0000 7f08bd14f640 1 -- 172.21.15.46:0/4278173264 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08b8108670 msgr2=0x7f08b8101900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:47.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.821+0000 7f08bd14f640 1 --2- 172.21.15.46:0/4278173264 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08b8108670 0x7f08b8101900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:47.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.821+0000 7f08bd14f640 1 -- 172.21.15.46:0/4278173264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08b8102190 con 0x7f08b8105d40 2024-04-26T21:49:47.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.821+0000 7f08bd950640 1 --2- 172.21.15.46:0/4278173264 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f08b8103410 0x7f08b8102d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:49:47.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.821+0000 7f08be151640 1 --2- 172.21.15.46:0/4278173264 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08b8108670 0x7f08b8101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:49:47.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.821+0000 7f08bd14f640 1 --2- 172.21.15.46:0/4278173264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08b8105d40 0x7f08b81013c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f08a800b450 tx=0x7f08a800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:47.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.821+0000 7f08aeffd640 1 -- 172.21.15.46:0/4278173264 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a8015020 con 0x7f08b8105d40 2024-04-26T21:49:47.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.821+0000 7f08aeffd640 1 -- 172.21.15.46:0/4278173264 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08a8011d40 con 0x7f08b8105d40 2024-04-26T21:49:47.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.821+0000 7f08aeffd640 1 -- 172.21.15.46:0/4278173264 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a800f340 con 0x7f08b8105d40 2024-04-26T21:49:47.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.821+0000 7f08bfbb4640 1 -- 172.21.15.46:0/4278173264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08b81a2d20 con 0x7f08b8105d40 2024-04-26T21:49:47.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.821+0000 7f08bfbb4640 1 -- 172.21.15.46:0/4278173264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08b81a31c0 con 0x7f08b8105d40 2024-04-26T21:49:47.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.822+0000 7f08bfbb4640 1 -- 172.21.15.46:0/4278173264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08b80fe910 con 0x7f08b8105d40 2024-04-26T21:49:47.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.824+0000 7f08aeffd640 1 -- 172.21.15.46:0/4278173264 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f08a8018020 con 0x7f08b8105d40 2024-04-26T21:49:47.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.824+0000 7f08aeffd640 1 --2- 172.21.15.46:0/4278173264 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f08a40a3c40 0x7f08a40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:47.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.824+0000 7f08bd950640 1 --2- 172.21.15.46:0/4278173264 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f08a40a3c40 0x7f08a40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:47.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.824+0000 7f08aeffd640 1 -- 172.21.15.46:0/4278173264 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f08a80d0580 con 0x7f08b8105d40 2024-04-26T21:49:47.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.825+0000 7f08bd950640 1 --2- 172.21.15.46:0/4278173264 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f08a40a3c40 0x7f08a40a6100 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f08b40058c0 tx=0x7f08b4005d30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:47.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:47.828+0000 7f08aeffd640 1 -- 172.21.15.46:0/4278173264 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f08a801f020 con 0x7f08b8105d40 2024-04-26T21:49:48.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.013+0000 7f08bfbb4640 1 -- 172.21.15.46:0/4278173264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f08b802dd60 con 0x7f08b8105d40 2024-04-26T21:49:48.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.013+0000 7f08aeffd640 1 -- 172.21.15.46:0/4278173264 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f08a800d040 con 0x7f08b8105d40 2024-04-26T21:49:48.014 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-04-26T21:49:48.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.015+0000 7f08bfbb4640 1 -- 172.21.15.46:0/4278173264 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f08a40a3c40 msgr2=0x7f08a40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:48.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.015+0000 7f08bfbb4640 1 --2- 172.21.15.46:0/4278173264 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f08a40a3c40 0x7f08a40a6100 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f08b40058c0 tx=0x7f08b4005d30 comp rx=0 tx=0).stop 2024-04-26T21:49:48.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.015+0000 7f08bfbb4640 1 -- 172.21.15.46:0/4278173264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08b8105d40 msgr2=0x7f08b81013c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:48.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.015+0000 7f08bfbb4640 1 --2- 172.21.15.46:0/4278173264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08b8105d40 0x7f08b81013c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f08a800b450 tx=0x7f08a800b910 comp rx=0 tx=0).stop 2024-04-26T21:49:48.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.015+0000 7f08bfbb4640 1 -- 172.21.15.46:0/4278173264 shutdown_connections 2024-04-26T21:49:48.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.015+0000 7f08bfbb4640 1 --2- 172.21.15.46:0/4278173264 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f08a40a3c40 0x7f08a40a6100 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.015+0000 7f08bfbb4640 1 --2- 172.21.15.46:0/4278173264 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f08b8108670 0x7f08b8101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.015+0000 7f08bfbb4640 1 --2- 172.21.15.46:0/4278173264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08b8105d40 0x7f08b81013c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.015+0000 7f08bfbb4640 1 --2- 172.21.15.46:0/4278173264 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f08b8103410 0x7f08b8102d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.015+0000 7f08bfbb4640 1 -- 172.21.15.46:0/4278173264 >> 172.21.15.46:0/4278173264 conn(0x7f08b80fce20 msgr2=0x7f08b8103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:48.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.016+0000 7f08bfbb4640 1 -- 172.21.15.46:0/4278173264 shutdown_connections 2024-04-26T21:49:48.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.016+0000 7f08bfbb4640 1 -- 172.21.15.46:0/4278173264 wait complete. 2024-04-26T21:49:48.026 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:47 smithi046 ceph-mon[29201]: pgmap v805: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:49:48.026 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:47 smithi046 ceph-mon[29201]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:49:48.027 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T21:49:48.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:47 smithi142 ceph-mon[32944]: pgmap v805: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:49:48.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:47 smithi142 ceph-mon[32944]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:49:48.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:47 smithi050 ceph-mon[32753]: pgmap v805: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:49:48.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:47 smithi050 ceph-mon[32753]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:49:48.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.131+0000 7f6e2a886640 1 -- 172.21.15.46:0/3302977006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e24101890 msgr2=0x7f6e24101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:48.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.131+0000 7f6e2a886640 1 --2- 172.21.15.46:0/3302977006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e24101890 0x7f6e24101cb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009900 tx=0x7f6e1402a410 comp rx=0 tx=0).stop 2024-04-26T21:49:48.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.131+0000 7f6e2a886640 1 -- 172.21.15.46:0/3302977006 shutdown_connections 2024-04-26T21:49:48.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.131+0000 7f6e2a886640 1 --2- 172.21.15.46:0/3302977006 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e24103440 0x7f6e24109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.131+0000 7f6e2a886640 1 --2- 172.21.15.46:0/3302977006 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e24102a80 0x7f6e24102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.131+0000 7f6e2a886640 1 --2- 172.21.15.46:0/3302977006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e24101890 0x7f6e24101cb0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.131+0000 7f6e2a886640 1 -- 172.21.15.46:0/3302977006 >> 172.21.15.46:0/3302977006 conn(0x7f6e240fce20 msgr2=0x7f6e240ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:48.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.131+0000 7f6e2a886640 1 -- 172.21.15.46:0/3302977006 shutdown_connections 2024-04-26T21:49:48.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.131+0000 7f6e2a886640 1 -- 172.21.15.46:0/3302977006 wait complete. 2024-04-26T21:49:48.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.131+0000 7f6e2a886640 1 Processor -- start 2024-04-26T21:49:48.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.132+0000 7f6e2a886640 1 -- start start 2024-04-26T21:49:48.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.132+0000 7f6e2a886640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e24101890 0x7f6e24198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:48.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.132+0000 7f6e2a886640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e24102a80 0x7f6e24199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.132+0000 7f6e2a886640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e24103440 0x7f6e241a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.132+0000 7f6e2a886640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2410d3c0 con 0x7f6e24103440 2024-04-26T21:49:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.132+0000 7f6e2a886640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2410d240 con 0x7f6e24101890 2024-04-26T21:49:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.132+0000 7f6e2a886640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2410d540 con 0x7f6e24102a80 2024-04-26T21:49:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.132+0000 7f6e28e23640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e24103440 0x7f6e241a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.132+0000 7f6e28e23640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e24103440 0x7f6e241a0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44754/0 (socket says 172.21.15.46:44754) 2024-04-26T21:49:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.132+0000 7f6e28e23640 1 -- 172.21.15.46:0/3815162806 learned_addr learned my addr 172.21.15.46:0/3815162806 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:49:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.132+0000 7f6e23fff640 1 --2- 172.21.15.46:0/3815162806 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e24101890 0x7f6e24198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.132+0000 7f6e237fe640 1 --2- 172.21.15.46:0/3815162806 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e24102a80 0x7f6e24199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.132+0000 7f6e28e23640 1 -- 172.21.15.46:0/3815162806 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e24102a80 msgr2=0x7f6e24199170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.132+0000 7f6e28e23640 1 --2- 172.21.15.46:0/3815162806 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e24102a80 0x7f6e24199170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.132+0000 7f6e28e23640 1 -- 172.21.15.46:0/3815162806 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e24101890 msgr2=0x7f6e24198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.132+0000 7f6e28e23640 1 --2- 172.21.15.46:0/3815162806 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e24101890 0x7f6e24198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.132+0000 7f6e28e23640 1 -- 172.21.15.46:0/3815162806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e241a0800 con 0x7f6e24103440 2024-04-26T21:49:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.133+0000 7f6e28e23640 1 --2- 172.21.15.46:0/3815162806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e24103440 0x7f6e241a0190 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c00eaa0 tx=0x7f6e1c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.133+0000 7f6e217fa640 1 -- 172.21.15.46:0/3815162806 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c00cdb0 con 0x7f6e24103440 2024-04-26T21:49:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.133+0000 7f6e217fa640 1 -- 172.21.15.46:0/3815162806 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e1c005020 con 0x7f6e24103440 2024-04-26T21:49:48.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.133+0000 7f6e2a886640 1 -- 172.21.15.46:0/3815162806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e241a0af0 con 0x7f6e24103440 2024-04-26T21:49:48.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.133+0000 7f6e217fa640 1 -- 172.21.15.46:0/3815162806 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c0116d0 con 0x7f6e24103440 2024-04-26T21:49:48.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.133+0000 7f6e2a886640 1 -- 172.21.15.46:0/3815162806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e241a1040 con 0x7f6e24103440 2024-04-26T21:49:48.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.134+0000 7f6e2a886640 1 -- 172.21.15.46:0/3815162806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e241a1270 con 0x7f6e24103440 2024-04-26T21:49:48.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.135+0000 7f6e217fa640 1 -- 172.21.15.46:0/3815162806 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6e1c0051d0 con 0x7f6e24103440 2024-04-26T21:49:48.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.136+0000 7f6e217fa640 1 --2- 172.21.15.46:0/3815162806 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6e0c0a3c40 0x7f6e0c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:48.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.136+0000 7f6e23fff640 1 --2- 172.21.15.46:0/3815162806 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6e0c0a3c40 0x7f6e0c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:48.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.136+0000 7f6e217fa640 1 -- 172.21.15.46:0/3815162806 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f6e1c0d0b40 con 0x7f6e24103440 2024-04-26T21:49:48.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.136+0000 7f6e23fff640 1 --2- 172.21.15.46:0/3815162806 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6e0c0a3c40 0x7f6e0c0a6100 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009230 tx=0x7f6e140314a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:48.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.139+0000 7f6e217fa640 1 -- 172.21.15.46:0/3815162806 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6e1c016030 con 0x7f6e24103440 2024-04-26T21:49:48.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.327+0000 7f6e2a886640 1 -- 172.21.15.46:0/3815162806 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6e241a1440 con 0x7f6e24103440 2024-04-26T21:49:48.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.327+0000 7f6e217fa640 1 -- 172.21.15.46:0/3815162806 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f6e1c09d390 con 0x7f6e24103440 2024-04-26T21:49:48.328 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-04-26T21:49:48.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.330+0000 7f6e2a886640 1 -- 172.21.15.46:0/3815162806 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6e0c0a3c40 msgr2=0x7f6e0c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:48.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.330+0000 7f6e2a886640 1 --2- 172.21.15.46:0/3815162806 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6e0c0a3c40 0x7f6e0c0a6100 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009230 tx=0x7f6e140314a0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.330+0000 7f6e2a886640 1 -- 172.21.15.46:0/3815162806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e24103440 msgr2=0x7f6e241a0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:48.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.330+0000 7f6e2a886640 1 --2- 172.21.15.46:0/3815162806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e24103440 0x7f6e241a0190 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c00eaa0 tx=0x7f6e1c00ef60 comp rx=0 tx=0).stop 2024-04-26T21:49:48.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.330+0000 7f6e2a886640 1 -- 172.21.15.46:0/3815162806 shutdown_connections 2024-04-26T21:49:48.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.330+0000 7f6e2a886640 1 --2- 172.21.15.46:0/3815162806 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f6e0c0a3c40 0x7f6e0c0a6100 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.330+0000 7f6e2a886640 1 --2- 172.21.15.46:0/3815162806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6e24103440 0x7f6e241a0190 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.330+0000 7f6e2a886640 1 --2- 172.21.15.46:0/3815162806 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f6e24102a80 0x7f6e24199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.330+0000 7f6e2a886640 1 --2- 172.21.15.46:0/3815162806 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e24101890 0x7f6e24198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.330+0000 7f6e2a886640 1 -- 172.21.15.46:0/3815162806 >> 172.21.15.46:0/3815162806 conn(0x7f6e240fce20 msgr2=0x7f6e2406af30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:48.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.330+0000 7f6e2a886640 1 -- 172.21.15.46:0/3815162806 shutdown_connections 2024-04-26T21:49:48.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.330+0000 7f6e2a886640 1 -- 172.21.15.46:0/3815162806 wait complete. 2024-04-26T21:49:48.341 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T21:49:48.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.445+0000 7f77279da640 1 -- 172.21.15.46:0/1905622935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7720108710 msgr2=0x7f772010ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:48.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.445+0000 7f77279da640 1 --2- 172.21.15.46:0/1905622935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7720108710 0x7f772010ab00 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f771c009ca0 tx=0x7f771c02a8c0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.445+0000 7f77279da640 1 -- 172.21.15.46:0/1905622935 shutdown_connections 2024-04-26T21:49:48.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.445+0000 7f77279da640 1 --2- 172.21.15.46:0/1905622935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7720108710 0x7f772010ab00 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.445+0000 7f77279da640 1 --2- 172.21.15.46:0/1905622935 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7720105de0 0x7f77201081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.445+0000 7f77279da640 1 --2- 172.21.15.46:0/1905622935 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f77201034b0 0x7f77201058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.445+0000 7f77279da640 1 -- 172.21.15.46:0/1905622935 >> 172.21.15.46:0/1905622935 conn(0x7f77200fcde0 msgr2=0x7f77200ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:48.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.446+0000 7f77279da640 1 -- 172.21.15.46:0/1905622935 shutdown_connections 2024-04-26T21:49:48.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.446+0000 7f77279da640 1 -- 172.21.15.46:0/1905622935 wait complete. 2024-04-26T21:49:48.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.446+0000 7f77279da640 1 Processor -- start 2024-04-26T21:49:48.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.446+0000 7f77279da640 1 -- start start 2024-04-26T21:49:48.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.446+0000 7f77279da640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77201034b0 0x7f7720198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:48.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.446+0000 7f77279da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7720105de0 0x7f7720199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:48.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.446+0000 7f77279da640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7720108710 0x7f77201a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:48.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.446+0000 7f77279da640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f772010e4d0 con 0x7f7720105de0 2024-04-26T21:49:48.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.446+0000 7f77279da640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f772010e350 con 0x7f77201034b0 2024-04-26T21:49:48.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.446+0000 7f77279da640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f772010e650 con 0x7f7720108710 2024-04-26T21:49:48.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.446+0000 7f7724f75640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7720105de0 0x7f7720199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:48.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.446+0000 7f7724f75640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7720105de0 0x7f7720199110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44760/0 (socket says 172.21.15.46:44760) 2024-04-26T21:49:48.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.446+0000 7f7724f75640 1 -- 172.21.15.46:0/10508175 learned_addr learned my addr 172.21.15.46:0/10508175 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:49:48.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.447+0000 7f7725f77640 1 --2- 172.21.15.46:0/10508175 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7720108710 0x7f77201a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:48.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.447+0000 7f7725776640 1 --2- 172.21.15.46:0/10508175 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77201034b0 0x7f7720198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:48.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.447+0000 7f7724f75640 1 -- 172.21.15.46:0/10508175 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7720108710 msgr2=0x7f77201a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:48.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.447+0000 7f7724f75640 1 --2- 172.21.15.46:0/10508175 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7720108710 0x7f77201a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.447+0000 7f7724f75640 1 -- 172.21.15.46:0/10508175 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77201034b0 msgr2=0x7f7720198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:48.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.447+0000 7f7724f75640 1 --2- 172.21.15.46:0/10508175 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77201034b0 0x7f7720198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.447+0000 7f7724f75640 1 -- 172.21.15.46:0/10508175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77201a07f0 con 0x7f7720105de0 2024-04-26T21:49:48.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.447+0000 7f7724f75640 1 --2- 172.21.15.46:0/10508175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7720105de0 0x7f7720199110 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7710005fd0 tx=0x7f771000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:48.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.447+0000 7f77167fc640 1 -- 172.21.15.46:0/10508175 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7710017070 con 0x7f7720105de0 2024-04-26T21:49:48.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.447+0000 7f77279da640 1 -- 172.21.15.46:0/10508175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77201a0ae0 con 0x7f7720105de0 2024-04-26T21:49:48.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.447+0000 7f77167fc640 1 -- 172.21.15.46:0/10508175 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7710009c30 con 0x7f7720105de0 2024-04-26T21:49:48.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.447+0000 7f77167fc640 1 -- 172.21.15.46:0/10508175 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7710003d30 con 0x7f7720105de0 2024-04-26T21:49:48.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.447+0000 7f77279da640 1 -- 172.21.15.46:0/10508175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77201a1080 con 0x7f7720105de0 2024-04-26T21:49:48.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.448+0000 7f77279da640 1 -- 172.21.15.46:0/10508175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77200fe8d0 con 0x7f7720105de0 2024-04-26T21:49:48.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.450+0000 7f77167fc640 1 -- 172.21.15.46:0/10508175 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7710003ed0 con 0x7f7720105de0 2024-04-26T21:49:48.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.450+0000 7f77167fc640 1 --2- 172.21.15.46:0/10508175 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f770c0a3c40 0x7f770c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:48.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.450+0000 7f77167fc640 1 -- 172.21.15.46:0/10508175 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f77100d0a30 con 0x7f7720105de0 2024-04-26T21:49:48.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.450+0000 7f7725776640 1 --2- 172.21.15.46:0/10508175 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f770c0a3c40 0x7f770c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:48.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.451+0000 7f7725776640 1 --2- 172.21.15.46:0/10508175 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f770c0a3c40 0x7f770c0a6100 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7718005fd0 tx=0x7f7718005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:48.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.454+0000 7f77167fc640 1 -- 172.21.15.46:0/10508175 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7710020030 con 0x7f7720105de0 2024-04-26T21:49:48.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.639+0000 7f77279da640 1 -- 172.21.15.46:0/10508175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f772002dad0 con 0x7f7720105de0 2024-04-26T21:49:48.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.640+0000 7f77167fc640 1 -- 172.21.15.46:0/10508175 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f7710009de0 con 0x7f7720105de0 2024-04-26T21:49:48.640 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-04-26T21:49:48.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.642+0000 7f77279da640 1 -- 172.21.15.46:0/10508175 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f770c0a3c40 msgr2=0x7f770c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:48.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.642+0000 7f77279da640 1 --2- 172.21.15.46:0/10508175 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f770c0a3c40 0x7f770c0a6100 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7718005fd0 tx=0x7f7718005850 comp rx=0 tx=0).stop 2024-04-26T21:49:48.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.642+0000 7f77279da640 1 -- 172.21.15.46:0/10508175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7720105de0 msgr2=0x7f7720199110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:48.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.642+0000 7f77279da640 1 --2- 172.21.15.46:0/10508175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7720105de0 0x7f7720199110 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7710005fd0 tx=0x7f771000dde0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.642+0000 7f77279da640 1 -- 172.21.15.46:0/10508175 shutdown_connections 2024-04-26T21:49:48.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.642+0000 7f77279da640 1 --2- 172.21.15.46:0/10508175 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f770c0a3c40 0x7f770c0a6100 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.642+0000 7f77279da640 1 --2- 172.21.15.46:0/10508175 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f7720108710 0x7f77201a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.642+0000 7f77279da640 1 --2- 172.21.15.46:0/10508175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7720105de0 0x7f7720199110 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.642+0000 7f77279da640 1 --2- 172.21.15.46:0/10508175 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f77201034b0 0x7f7720198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.642+0000 7f77279da640 1 -- 172.21.15.46:0/10508175 >> 172.21.15.46:0/10508175 conn(0x7f77200fcde0 msgr2=0x7f7720103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:48.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.642+0000 7f77279da640 1 -- 172.21.15.46:0/10508175 shutdown_connections 2024-04-26T21:49:48.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.642+0000 7f77279da640 1 -- 172.21.15.46:0/10508175 wait complete. 2024-04-26T21:49:48.654 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T21:49:48.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.756+0000 7fcca8c2c640 1 -- 172.21.15.46:0/3247774965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcca4103410 msgr2=0x7fcca4105800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:48.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.756+0000 7fcca8c2c640 1 --2- 172.21.15.46:0/3247774965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcca4103410 0x7fcca4105800 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fcc94009900 tx=0x7fcc9402a410 comp rx=0 tx=0).stop 2024-04-26T21:49:48.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.756+0000 7fcca8c2c640 1 -- 172.21.15.46:0/3247774965 shutdown_connections 2024-04-26T21:49:48.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.756+0000 7fcca8c2c640 1 --2- 172.21.15.46:0/3247774965 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcca4108670 0x7fcca410aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.756+0000 7fcca8c2c640 1 --2- 172.21.15.46:0/3247774965 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcca4105d40 0x7fcca4108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.756+0000 7fcca8c2c640 1 --2- 172.21.15.46:0/3247774965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcca4103410 0x7fcca4105800 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.756+0000 7fcca8c2c640 1 -- 172.21.15.46:0/3247774965 >> 172.21.15.46:0/3247774965 conn(0x7fcca40fce20 msgr2=0x7fcca40ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:48.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.756+0000 7fcca8c2c640 1 -- 172.21.15.46:0/3247774965 shutdown_connections 2024-04-26T21:49:48.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.756+0000 7fcca8c2c640 1 -- 172.21.15.46:0/3247774965 wait complete. 2024-04-26T21:49:48.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.757+0000 7fcca8c2c640 1 Processor -- start 2024-04-26T21:49:48.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.757+0000 7fcca8c2c640 1 -- start start 2024-04-26T21:49:48.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.757+0000 7fcca8c2c640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcca4103410 0x7fcca419ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:48.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.757+0000 7fcca8c2c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcca4105d40 0x7fcca419b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:48.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.757+0000 7fcca8c2c640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcca4108670 0x7fcca41a23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:48.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.757+0000 7fcca8c2c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fcca410e4c0 con 0x7fcca4105d40 2024-04-26T21:49:48.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.757+0000 7fcca8c2c640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcca410e340 con 0x7fcca4103410 2024-04-26T21:49:48.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.757+0000 7fcca8c2c640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fcca410e640 con 0x7fcca4108670 2024-04-26T21:49:48.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.757+0000 7fcc9bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcca4105d40 0x7fcca419b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:48.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.757+0000 7fcc9bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcca4105d40 0x7fcca419b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44776/0 (socket says 172.21.15.46:44776) 2024-04-26T21:49:48.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.757+0000 7fcc9bfff640 1 -- 172.21.15.46:0/1195183115 learned_addr learned my addr 172.21.15.46:0/1195183115 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:49:48.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.757+0000 7fcca2d9d640 1 --2- 172.21.15.46:0/1195183115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcca4108670 0x7fcca41a23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:48.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.757+0000 7fcca259c640 1 --2- 172.21.15.46:0/1195183115 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcca4103410 0x7fcca419ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:48.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.757+0000 7fcc9bfff640 1 -- 172.21.15.46:0/1195183115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcca4108670 msgr2=0x7fcca41a23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:48.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.757+0000 7fcc9bfff640 1 --2- 172.21.15.46:0/1195183115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcca4108670 0x7fcca41a23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.757+0000 7fcc9bfff640 1 -- 172.21.15.46:0/1195183115 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcca4103410 msgr2=0x7fcca419ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:48.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.757+0000 7fcc9bfff640 1 --2- 172.21.15.46:0/1195183115 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcca4103410 0x7fcca419ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.757+0000 7fcc9bfff640 1 -- 172.21.15.46:0/1195183115 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcca41a2a30 con 0x7fcca4105d40 2024-04-26T21:49:48.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.758+0000 7fcc9bfff640 1 --2- 172.21.15.46:0/1195183115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcca4105d40 0x7fcca419b350 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c00d770 tx=0x7fcc8c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:48.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.758+0000 7fcc9b7fe640 1 -- 172.21.15.46:0/1195183115 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c017070 con 0x7fcca4105d40 2024-04-26T21:49:48.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.758+0000 7fcc9b7fe640 1 -- 172.21.15.46:0/1195183115 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc8c009c30 con 0x7fcca4105d40 2024-04-26T21:49:48.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.758+0000 7fcca8c2c640 1 -- 172.21.15.46:0/1195183115 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcca41a2d20 con 0x7fcca4105d40 2024-04-26T21:49:48.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.758+0000 7fcc9b7fe640 1 -- 172.21.15.46:0/1195183115 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c00f620 con 0x7fcca4105d40 2024-04-26T21:49:48.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.758+0000 7fcca8c2c640 1 -- 172.21.15.46:0/1195183115 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcca41a3260 con 0x7fcca4105d40 2024-04-26T21:49:48.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.759+0000 7fcca8c2c640 1 -- 172.21.15.46:0/1195183115 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcca410a1c0 con 0x7fcca4105d40 2024-04-26T21:49:48.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.760+0000 7fcc9b7fe640 1 -- 172.21.15.46:0/1195183115 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fcc8c01a070 con 0x7fcca4105d40 2024-04-26T21:49:48.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.761+0000 7fcc9b7fe640 1 --2- 172.21.15.46:0/1195183115 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcc900a3cc0 0x7fcc900a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:48.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.761+0000 7fcca259c640 1 --2- 172.21.15.46:0/1195183115 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcc900a3cc0 0x7fcc900a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:48.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.761+0000 7fcc9b7fe640 1 -- 172.21.15.46:0/1195183115 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7fcc8c0d0130 con 0x7fcca4105d40 2024-04-26T21:49:48.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.761+0000 7fcca259c640 1 --2- 172.21.15.46:0/1195183115 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcc900a3cc0 0x7fcc900a6180 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcc94000c00 tx=0x7fcc940314a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:48.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.765+0000 7fcc9b7fe640 1 -- 172.21.15.46:0/1195183115 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcc8c009de0 con 0x7fcca4105d40 2024-04-26T21:49:48.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.967+0000 7fcca8c2c640 1 -- 172.21.15.46:0/1195183115 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fcca402dad0 con 0x7fcca4105d40 2024-04-26T21:49:48.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.968+0000 7fcc9b7fe640 1 -- 172.21.15.46:0/1195183115 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v87) ==== 81+0+592 (secure 0 0 0) 0x7fcc8c09c980 con 0x7fcca4105d40 2024-04-26T21:49:48.969 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/2465242088 2024-04-27T21:25:29.848343+0000 2024-04-26T21:49:48.969 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6801/2053782984 2024-04-27T21:25:29.848343+0000 2024-04-26T21:49:48.969 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6800/2053782984 2024-04-27T21:25:29.848343+0000 2024-04-26T21:49:48.969 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/1381997090 2024-04-27T21:25:02.078730+0000 2024-04-26T21:49:48.969 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/3507268217 2024-04-27T21:25:29.848343+0000 2024-04-26T21:49:48.969 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/2570024356 2024-04-27T21:25:02.078730+0000 2024-04-26T21:49:48.969 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/2691061938 2024-04-27T21:25:29.848343+0000 2024-04-26T21:49:48.970 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/2775954801 2024-04-27T21:25:02.078730+0000 2024-04-26T21:49:48.970 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6800/2896098005 2024-04-27T21:25:02.078730+0000 2024-04-26T21:49:48.970 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6801/2896098005 2024-04-27T21:25:02.078730+0000 2024-04-26T21:49:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:listed 10 entries 2024-04-26T21:49:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.970+0000 7fcca8c2c640 1 -- 172.21.15.46:0/1195183115 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcc900a3cc0 msgr2=0x7fcc900a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.970+0000 7fcca8c2c640 1 --2- 172.21.15.46:0/1195183115 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcc900a3cc0 0x7fcc900a6180 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcc94000c00 tx=0x7fcc940314a0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.970+0000 7fcca8c2c640 1 -- 172.21.15.46:0/1195183115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcca4105d40 msgr2=0x7fcca419b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.970+0000 7fcca8c2c640 1 --2- 172.21.15.46:0/1195183115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcca4105d40 0x7fcca419b350 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c00d770 tx=0x7fcc8c00dc30 comp rx=0 tx=0).stop 2024-04-26T21:49:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.970+0000 7fcca8c2c640 1 -- 172.21.15.46:0/1195183115 shutdown_connections 2024-04-26T21:49:48.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.970+0000 7fcca8c2c640 1 --2- 172.21.15.46:0/1195183115 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fcc900a3cc0 0x7fcc900a6180 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.970+0000 7fcca8c2c640 1 --2- 172.21.15.46:0/1195183115 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fcca4108670 0x7fcca41a23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.970+0000 7fcca8c2c640 1 --2- 172.21.15.46:0/1195183115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcca4105d40 0x7fcca419b350 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.970+0000 7fcca8c2c640 1 --2- 172.21.15.46:0/1195183115 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcca4103410 0x7fcca419ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:48.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.970+0000 7fcca8c2c640 1 -- 172.21.15.46:0/1195183115 >> 172.21.15.46:0/1195183115 conn(0x7fcca40fce20 msgr2=0x7fcca4102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:48.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.970+0000 7fcca8c2c640 1 -- 172.21.15.46:0/1195183115 shutdown_connections 2024-04-26T21:49:48.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:48.970+0000 7fcca8c2c640 1 -- 172.21.15.46:0/1195183115 wait complete. 2024-04-26T21:49:48.980 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:48 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3399598926' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:49:48.980 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:48 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/4278173264' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:49:48.980 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:48 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3815162806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:49:48.980 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:48 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/10508175' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:49:48.981 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-26T21:49:48.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:49:48.982 DEBUG:teuthology.orchestra.run.smithi046:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-26T21:49:49.012 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-04-26T21:49:49.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:49:49.012 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-26T21:49:49.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:48 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3399598926' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:49:49.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:48 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/4278173264' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:49:49.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:48 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3815162806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:49:49.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:48 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/10508175' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:49:49.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:49:49.084 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-04-26T21:49:49.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:48 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3399598926' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:49:49.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:48 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/4278173264' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:49:49.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:48 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3815162806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:49:49.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:48 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/10508175' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:49:49.139 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-26T21:49:49.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:49:49.140 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-26T21:49:49.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:49:49.205 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link delete ceph-brx 2024-04-26T21:49:49.289 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-26T21:49:49.299 INFO:tasks.cephadm:Teardown begin 2024-04-26T21:49:49.299 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-26T21:49:49.299 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:49:49.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.432+0000 7f8946d78640 1 -- 172.21.15.46:0/1919122628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8940102a60 msgr2=0x7f8940102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:49.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.432+0000 7f8946d78640 1 --2- 172.21.15.46:0/1919122628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8940102a60 0x7f8940102ee0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f892c009900 tx=0x7f892c02a410 comp rx=0 tx=0).stop 2024-04-26T21:49:49.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.432+0000 7f8946d78640 1 -- 172.21.15.46:0/1919122628 shutdown_connections 2024-04-26T21:49:49.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.432+0000 7f8946d78640 1 --2- 172.21.15.46:0/1919122628 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8940103420 0x7f8940109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:49.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.432+0000 7f8946d78640 1 --2- 172.21.15.46:0/1919122628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8940102a60 0x7f8940102ee0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:49.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.432+0000 7f8946d78640 1 --2- 172.21.15.46:0/1919122628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8940101870 0x7f8940101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:49.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.432+0000 7f8946d78640 1 -- 172.21.15.46:0/1919122628 >> 172.21.15.46:0/1919122628 conn(0x7f89400fcde0 msgr2=0x7f89400ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:49.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.432+0000 7f8946d78640 1 -- 172.21.15.46:0/1919122628 shutdown_connections 2024-04-26T21:49:49.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.432+0000 7f8946d78640 1 -- 172.21.15.46:0/1919122628 wait complete. 2024-04-26T21:49:49.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.432+0000 7f8946d78640 1 Processor -- start 2024-04-26T21:49:49.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.433+0000 7f8946d78640 1 -- start start 2024-04-26T21:49:49.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.433+0000 7f8946d78640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8940101870 0x7f8940198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:49.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.433+0000 7f8946d78640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8940102a60 0x7f8940199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:49.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.433+0000 7f8946d78640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8940103420 0x7f89401a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:49.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.433+0000 7f8946d78640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f894010d380 con 0x7f8940101870 2024-04-26T21:49:49.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.433+0000 7f8946d78640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f894010d200 con 0x7f8940103420 2024-04-26T21:49:49.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.433+0000 7f8946d78640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f894010d500 con 0x7f8940102a60 2024-04-26T21:49:49.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.433+0000 7f8944b14640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8940101870 0x7f8940198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:49.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.433+0000 7f8944b14640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8940101870 0x7f8940198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44796/0 (socket says 172.21.15.46:44796) 2024-04-26T21:49:49.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.433+0000 7f8944b14640 1 -- 172.21.15.46:0/2947381884 learned_addr learned my addr 172.21.15.46:0/2947381884 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:49:49.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.433+0000 7f893ffff640 1 --2- 172.21.15.46:0/2947381884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8940102a60 0x7f8940199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:49.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.433+0000 7f8945315640 1 --2- 172.21.15.46:0/2947381884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8940103420 0x7f89401a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:49.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.433+0000 7f8944b14640 1 -- 172.21.15.46:0/2947381884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8940102a60 msgr2=0x7f8940199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:49.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.433+0000 7f8944b14640 1 --2- 172.21.15.46:0/2947381884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8940102a60 0x7f8940199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:49.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.433+0000 7f8944b14640 1 -- 172.21.15.46:0/2947381884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8940103420 msgr2=0x7f89401a0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:49.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.433+0000 7f8944b14640 1 --2- 172.21.15.46:0/2947381884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8940103420 0x7f89401a0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:49.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.433+0000 7f8944b14640 1 -- 172.21.15.46:0/2947381884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89401a0830 con 0x7f8940101870 2024-04-26T21:49:49.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.433+0000 7f8945315640 1 --2- 172.21.15.46:0/2947381884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8940103420 0x7f89401a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:49:49.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.433+0000 7f8944b14640 1 --2- 172.21.15.46:0/2947381884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8940101870 0x7f8940198c20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f893400e9c0 tx=0x7f893400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:49.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.433+0000 7f893dffb640 1 -- 172.21.15.46:0/2947381884 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f893400cd80 con 0x7f8940101870 2024-04-26T21:49:49.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.434+0000 7f893dffb640 1 -- 172.21.15.46:0/2947381884 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89340035f0 con 0x7f8940101870 2024-04-26T21:49:49.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.434+0000 7f8946d78640 1 -- 172.21.15.46:0/2947381884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89401a0b20 con 0x7f8940101870 2024-04-26T21:49:49.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.434+0000 7f893dffb640 1 -- 172.21.15.46:0/2947381884 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8934010690 con 0x7f8940101870 2024-04-26T21:49:49.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.434+0000 7f8946d78640 1 -- 172.21.15.46:0/2947381884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89401a1070 con 0x7f8940101870 2024-04-26T21:49:49.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.435+0000 7f8946d78640 1 -- 172.21.15.46:0/2947381884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89401a1240 con 0x7f8940101870 2024-04-26T21:49:49.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.436+0000 7f893dffb640 1 -- 172.21.15.46:0/2947381884 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f89340037a0 con 0x7f8940101870 2024-04-26T21:49:49.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.437+0000 7f893dffb640 1 --2- 172.21.15.46:0/2947381884 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f89300a3c40 0x7f89300a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:49.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.437+0000 7f893ffff640 1 --2- 172.21.15.46:0/2947381884 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f89300a3c40 0x7f89300a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:49.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.437+0000 7f893dffb640 1 -- 172.21.15.46:0/2947381884 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f89340d09e0 con 0x7f8940101870 2024-04-26T21:49:49.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.437+0000 7f893ffff640 1 --2- 172.21.15.46:0/2947381884 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f89300a3c40 0x7f89300a6100 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f892c000c00 tx=0x7f892c0314a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:49.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.441+0000 7f893dffb640 1 -- 172.21.15.46:0/2947381884 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f893409d290 con 0x7f8940101870 2024-04-26T21:49:49.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.670+0000 7f8946d78640 1 -- 172.21.15.46:0/2947381884 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89401a1410 con 0x7f8940101870 2024-04-26T21:49:49.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.670+0000 7f893dffb640 1 -- 172.21.15.46:0/2947381884 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11863 (secure 0 0 0) 0x7f893409cb90 con 0x7f8940101870 2024-04-26T21:49:49.671 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:49:49.671 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3674296286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3674296286},{"type":"v1","addr":"172.21.15.46:6835","nonce":3674296286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/391791206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":391791206},{"type":"v1","addr":"172.21.15.46:6837","nonce":391791206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/1222489912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1222489912},{"type":"v1","addr":"172.21.15.46:6839","nonce":1222489912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14658,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/860605548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":860605548},{"type":"v1","addr":"172.21.15.46:6841","nonce":860605548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3621399933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3621399933},{"type":"v1","addr":"172.21.15.142:6833","nonce":3621399933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/345015778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":345015778},{"type":"v1","addr":"172.21.15.50:6833","nonce":345015778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T21:32:44.201541+0000","modified":"2024-04-26T21:34:46.555334+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24526,"mds_2":24509},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/2546509863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":2546509863},{"type":"v1","addr":"172.21.15.50:6835","nonce":2546509863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/3791468066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":3791468066},{"type":"v1","addr":"172.21.15.142:6835","nonce":3791468066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6837/1723132553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1723132553},{"type":"v1","addr":"172.21.15.142:6837","nonce":1723132553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6837/3189620395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3189620395},{"type":"v1","addr":"172.21.15.50:6837","nonce":3189620395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24526":{"gid":24526,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.142:6839/3089509101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3089509101},{"type":"v1","addr":"172.21.15.142:6839","nonce":3089509101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6839/2453253368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2453253368},{"type":"v1","addr":"172.21.15.50:6839","nonce":2453253368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24539,24509,24526]},"id":1}]} 2024-04-26T21:49:49.672 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 23 2024-04-26T21:49:49.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.673+0000 7f8946d78640 1 -- 172.21.15.46:0/2947381884 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f89300a3c40 msgr2=0x7f89300a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:49.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.673+0000 7f8946d78640 1 --2- 172.21.15.46:0/2947381884 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f89300a3c40 0x7f89300a6100 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f892c000c00 tx=0x7f892c0314a0 comp rx=0 tx=0).stop 2024-04-26T21:49:49.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.673+0000 7f8946d78640 1 -- 172.21.15.46:0/2947381884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8940101870 msgr2=0x7f8940198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:49.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.673+0000 7f8946d78640 1 --2- 172.21.15.46:0/2947381884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8940101870 0x7f8940198c20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f893400e9c0 tx=0x7f893400ee80 comp rx=0 tx=0).stop 2024-04-26T21:49:49.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.673+0000 7f8946d78640 1 -- 172.21.15.46:0/2947381884 shutdown_connections 2024-04-26T21:49:49.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.673+0000 7f8946d78640 1 --2- 172.21.15.46:0/2947381884 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f89300a3c40 0x7f89300a6100 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:49.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.673+0000 7f8946d78640 1 --2- 172.21.15.46:0/2947381884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8940103420 0x7f89401a0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:49.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.673+0000 7f8946d78640 1 --2- 172.21.15.46:0/2947381884 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f8940102a60 0x7f8940199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:49.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.673+0000 7f8946d78640 1 --2- 172.21.15.46:0/2947381884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8940101870 0x7f8940198c20 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:49.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.673+0000 7f8946d78640 1 -- 172.21.15.46:0/2947381884 >> 172.21.15.46:0/2947381884 conn(0x7f89400fcde0 msgr2=0x7f8940069b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:49.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.673+0000 7f8946d78640 1 -- 172.21.15.46:0/2947381884 shutdown_connections 2024-04-26T21:49:49.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.673+0000 7f8946d78640 1 -- 172.21.15.46:0/2947381884 wait complete. 2024-04-26T21:49:49.684 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:49:49.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.785+0000 7f0b57119640 1 -- 172.21.15.46:0/2299951271 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b50102a80 msgr2=0x7f0b50102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:49.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.785+0000 7f0b57119640 1 --2- 172.21.15.46:0/2299951271 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b50102a80 0x7f0b50102f00 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c009900 tx=0x7f0b3c02a410 comp rx=0 tx=0).stop 2024-04-26T21:49:49.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.785+0000 7f0b57119640 1 -- 172.21.15.46:0/2299951271 shutdown_connections 2024-04-26T21:49:49.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.785+0000 7f0b57119640 1 --2- 172.21.15.46:0/2299951271 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b50103440 0x7f0b50109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:49.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.785+0000 7f0b57119640 1 --2- 172.21.15.46:0/2299951271 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b50102a80 0x7f0b50102f00 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:49.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.785+0000 7f0b57119640 1 --2- 172.21.15.46:0/2299951271 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b50101890 0x7f0b50101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:49.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.785+0000 7f0b57119640 1 -- 172.21.15.46:0/2299951271 >> 172.21.15.46:0/2299951271 conn(0x7f0b500fce20 msgr2=0x7f0b500ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:49.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.785+0000 7f0b57119640 1 -- 172.21.15.46:0/2299951271 shutdown_connections 2024-04-26T21:49:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.785+0000 7f0b57119640 1 -- 172.21.15.46:0/2299951271 wait complete. 2024-04-26T21:49:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.786+0000 7f0b57119640 1 Processor -- start 2024-04-26T21:49:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.786+0000 7f0b57119640 1 -- start start 2024-04-26T21:49:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.786+0000 7f0b57119640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b50101890 0x7f0b50198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.786+0000 7f0b57119640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b50102a80 0x7f0b50199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.786+0000 7f0b57119640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b50103440 0x7f0b501a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.786+0000 7f0b57119640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0b5010d3a0 con 0x7f0b50101890 2024-04-26T21:49:49.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.786+0000 7f0b57119640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0b5010d220 con 0x7f0b50103440 2024-04-26T21:49:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.786+0000 7f0b57119640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0b5010d520 con 0x7f0b50102a80 2024-04-26T21:49:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.786+0000 7f0b54eb5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b50101890 0x7f0b50198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.786+0000 7f0b54eb5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b50101890 0x7f0b50198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44810/0 (socket says 172.21.15.46:44810) 2024-04-26T21:49:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.786+0000 7f0b54eb5640 1 -- 172.21.15.46:0/1290160843 learned_addr learned my addr 172.21.15.46:0/1290160843 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:49:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.786+0000 7f0b556b6640 1 --2- 172.21.15.46:0/1290160843 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b50103440 0x7f0b501a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.786+0000 7f0b4ffff640 1 --2- 172.21.15.46:0/1290160843 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b50102a80 0x7f0b50199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.786+0000 7f0b54eb5640 1 -- 172.21.15.46:0/1290160843 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b50102a80 msgr2=0x7f0b50199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:49.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.786+0000 7f0b54eb5640 1 --2- 172.21.15.46:0/1290160843 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b50102a80 0x7f0b50199110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.786+0000 7f0b54eb5640 1 -- 172.21.15.46:0/1290160843 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b50103440 msgr2=0x7f0b501a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.786+0000 7f0b54eb5640 1 --2- 172.21.15.46:0/1290160843 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b50103440 0x7f0b501a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.786+0000 7f0b54eb5640 1 -- 172.21.15.46:0/1290160843 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b501a07f0 con 0x7f0b50101890 2024-04-26T21:49:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.787+0000 7f0b556b6640 1 --2- 172.21.15.46:0/1290160843 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b50103440 0x7f0b501a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:49:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.787+0000 7f0b54eb5640 1 --2- 172.21.15.46:0/1290160843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b50101890 0x7f0b50198bd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0b4400e9c0 tx=0x7f0b4400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.787+0000 7f0b4dffb640 1 -- 172.21.15.46:0/1290160843 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4400cd80 con 0x7f0b50101890 2024-04-26T21:49:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.787+0000 7f0b4dffb640 1 -- 172.21.15.46:0/1290160843 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b440035f0 con 0x7f0b50101890 2024-04-26T21:49:49.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.787+0000 7f0b4dffb640 1 -- 172.21.15.46:0/1290160843 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b44010690 con 0x7f0b50101890 2024-04-26T21:49:49.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.787+0000 7f0b57119640 1 -- 172.21.15.46:0/1290160843 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b501a0ae0 con 0x7f0b50101890 2024-04-26T21:49:49.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.787+0000 7f0b57119640 1 -- 172.21.15.46:0/1290160843 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b501a1080 con 0x7f0b50101890 2024-04-26T21:49:49.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.788+0000 7f0b57119640 1 -- 172.21.15.46:0/1290160843 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b501a12b0 con 0x7f0b50101890 2024-04-26T21:49:49.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.789+0000 7f0b4dffb640 1 -- 172.21.15.46:0/1290160843 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0b440037a0 con 0x7f0b50101890 2024-04-26T21:49:49.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.790+0000 7f0b4dffb640 1 --2- 172.21.15.46:0/1290160843 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f0b400a3cc0 0x7f0b400a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:49.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.790+0000 7f0b4dffb640 1 -- 172.21.15.46:0/1290160843 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f0b440d09c0 con 0x7f0b50101890 2024-04-26T21:49:49.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.790+0000 7f0b4ffff640 1 --2- 172.21.15.46:0/1290160843 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f0b400a3cc0 0x7f0b400a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:49.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.790+0000 7f0b4ffff640 1 --2- 172.21.15.46:0/1290160843 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f0b400a3cc0 0x7f0b400a6180 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c02a930 tx=0x7f0b3c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:49.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:49.795+0000 7f0b4dffb640 1 -- 172.21.15.46:0/1290160843 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0b4409d270 con 0x7f0b50101890 2024-04-26T21:49:49.941 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:49 smithi142 ceph-mon[32944]: pgmap v806: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:49.941 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:49 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/1195183115' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T21:49:49.941 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:49 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/2947381884' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:49:50.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.019+0000 7f0b57119640 1 -- 172.21.15.46:0/1290160843 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b501a1480 con 0x7f0b50101890 2024-04-26T21:49:50.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.019+0000 7f0b4dffb640 1 -- 172.21.15.46:0/1290160843 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11863 (secure 0 0 0) 0x7f0b4409cb70 con 0x7f0b50101890 2024-04-26T21:49:50.020 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:49:50.020 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/3674296286","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":3674296286},{"type":"v1","addr":"172.21.15.46:6835","nonce":3674296286}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/391791206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":391791206},{"type":"v1","addr":"172.21.15.46:6837","nonce":391791206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14640,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/1222489912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":1222489912},{"type":"v1","addr":"172.21.15.46:6839","nonce":1222489912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14658,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/860605548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":860605548},{"type":"v1","addr":"172.21.15.46:6841","nonce":860605548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.142:6833/3621399933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6832","nonce":3621399933},{"type":"v1","addr":"172.21.15.142:6833","nonce":3621399933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.50:6833/345015778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":345015778},{"type":"v1","addr":"172.21.15.50:6833","nonce":345015778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T21:32:44.201541+0000","modified":"2024-04-26T21:34:46.555334+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24526,"mds_2":24509},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24470":{"gid":24470,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.50:6835/2546509863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6834","nonce":2546509863},{"type":"v1","addr":"172.21.15.50:6835","nonce":2546509863}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24493":{"gid":24493,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6835/3791468066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6834","nonce":3791468066},{"type":"v1","addr":"172.21.15.142:6835","nonce":3791468066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.142:6837/1723132553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6836","nonce":1723132553},{"type":"v1","addr":"172.21.15.142:6837","nonce":1723132553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.50:6837/3189620395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6836","nonce":3189620395},{"type":"v1","addr":"172.21.15.50:6837","nonce":3189620395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24526":{"gid":24526,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.142:6839/3089509101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6838","nonce":3089509101},{"type":"v1","addr":"172.21.15.142:6839","nonce":3089509101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.50:6839/2453253368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6838","nonce":2453253368},{"type":"v1","addr":"172.21.15.50:6839","nonce":2453253368}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24539,24509,24526]},"id":1}]} 2024-04-26T21:49:50.020 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 23 2024-04-26T21:49:50.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.021+0000 7f0b57119640 1 -- 172.21.15.46:0/1290160843 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f0b400a3cc0 msgr2=0x7f0b400a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:50.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.021+0000 7f0b57119640 1 --2- 172.21.15.46:0/1290160843 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f0b400a3cc0 0x7f0b400a6180 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c02a930 tx=0x7f0b3c000bc0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.021+0000 7f0b57119640 1 -- 172.21.15.46:0/1290160843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b50101890 msgr2=0x7f0b50198bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:50.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.021+0000 7f0b57119640 1 --2- 172.21.15.46:0/1290160843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b50101890 0x7f0b50198bd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0b4400e9c0 tx=0x7f0b4400ee80 comp rx=0 tx=0).stop 2024-04-26T21:49:50.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.021+0000 7f0b57119640 1 -- 172.21.15.46:0/1290160843 shutdown_connections 2024-04-26T21:49:50.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.021+0000 7f0b57119640 1 --2- 172.21.15.46:0/1290160843 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f0b400a3cc0 0x7f0b400a6180 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.021+0000 7f0b57119640 1 --2- 172.21.15.46:0/1290160843 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b50103440 0x7f0b501a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.021+0000 7f0b57119640 1 --2- 172.21.15.46:0/1290160843 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0b50102a80 0x7f0b50199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.021+0000 7f0b57119640 1 --2- 172.21.15.46:0/1290160843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b50101890 0x7f0b50198bd0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.021+0000 7f0b57119640 1 -- 172.21.15.46:0/1290160843 >> 172.21.15.46:0/1290160843 conn(0x7f0b500fce20 msgr2=0x7f0b5006af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:50.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.021+0000 7f0b57119640 1 -- 172.21.15.46:0/1290160843 shutdown_connections 2024-04-26T21:49:50.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.021+0000 7f0b57119640 1 -- 172.21.15.46:0/1290160843 wait complete. 2024-04-26T21:49:50.031 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:49 smithi046 ceph-mon[29201]: pgmap v806: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:50.032 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:49 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1195183115' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T21:49:50.032 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:49 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2947381884' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:49:50.033 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:49:50.117 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:49 smithi050 ceph-mon[32753]: pgmap v806: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:49:50.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:49 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/1195183115' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T21:49:50.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:49 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/2947381884' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:49:50.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.135+0000 7f4e5f7f0640 1 -- 172.21.15.46:0/1743864755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e58102a80 msgr2=0x7f4e58102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:50.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.135+0000 7f4e5f7f0640 1 --2- 172.21.15.46:0/1743864755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e58102a80 0x7f4e58102f00 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4e48009900 tx=0x7f4e4802a410 comp rx=0 tx=0).stop 2024-04-26T21:49:50.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.135+0000 7f4e5f7f0640 1 -- 172.21.15.46:0/1743864755 shutdown_connections 2024-04-26T21:49:50.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.135+0000 7f4e5f7f0640 1 --2- 172.21.15.46:0/1743864755 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e58103440 0x7f4e58109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.135+0000 7f4e5f7f0640 1 --2- 172.21.15.46:0/1743864755 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e58102a80 0x7f4e58102f00 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.135+0000 7f4e5f7f0640 1 --2- 172.21.15.46:0/1743864755 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e58101890 0x7f4e58101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.135+0000 7f4e5f7f0640 1 -- 172.21.15.46:0/1743864755 >> 172.21.15.46:0/1743864755 conn(0x7f4e580fce20 msgr2=0x7f4e580ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:50.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.135+0000 7f4e5f7f0640 1 -- 172.21.15.46:0/1743864755 shutdown_connections 2024-04-26T21:49:50.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.135+0000 7f4e5f7f0640 1 -- 172.21.15.46:0/1743864755 wait complete. 2024-04-26T21:49:50.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.136+0000 7f4e5f7f0640 1 Processor -- start 2024-04-26T21:49:50.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.136+0000 7f4e5f7f0640 1 -- start start 2024-04-26T21:49:50.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.136+0000 7f4e5f7f0640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e58101890 0x7f4e58197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:50.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.136+0000 7f4e5f7f0640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e58102a80 0x7f4e58197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:50.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.136+0000 7f4e5f7f0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e58103440 0x7f4e5819f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:50.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.136+0000 7f4e5f7f0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f4e5810d3f0 con 0x7f4e58103440 2024-04-26T21:49:50.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.136+0000 7f4e5f7f0640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4e5810d270 con 0x7f4e58101890 2024-04-26T21:49:50.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.136+0000 7f4e5f7f0640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f4e5810d570 con 0x7f4e58102a80 2024-04-26T21:49:50.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.136+0000 7f4e5dd8d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e58103440 0x7f4e5819f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:50.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.136+0000 7f4e5dd8d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e58103440 0x7f4e5819f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44832/0 (socket says 172.21.15.46:44832) 2024-04-26T21:49:50.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.136+0000 7f4e5dd8d640 1 -- 172.21.15.46:0/2181468062 learned_addr learned my addr 172.21.15.46:0/2181468062 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:49:50.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.136+0000 7f4e5d58c640 1 --2- 172.21.15.46:0/2181468062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e58101890 0x7f4e58197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:50.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.136+0000 7f4e5cd8b640 1 --2- 172.21.15.46:0/2181468062 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e58102a80 0x7f4e58197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:50.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.136+0000 7f4e5dd8d640 1 -- 172.21.15.46:0/2181468062 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e58102a80 msgr2=0x7f4e58197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:50.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.136+0000 7f4e5dd8d640 1 --2- 172.21.15.46:0/2181468062 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e58102a80 0x7f4e58197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.136+0000 7f4e5dd8d640 1 -- 172.21.15.46:0/2181468062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e58101890 msgr2=0x7f4e58197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:50.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.136+0000 7f4e5dd8d640 1 --2- 172.21.15.46:0/2181468062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e58101890 0x7f4e58197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.136+0000 7f4e5dd8d640 1 -- 172.21.15.46:0/2181468062 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e5819f680 con 0x7f4e58103440 2024-04-26T21:49:50.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.136+0000 7f4e5dd8d640 1 --2- 172.21.15.46:0/2181468062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e58103440 0x7f4e5819f010 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4e50007900 tx=0x7f4e50007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:50.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.137+0000 7f4e4e7fc640 1 -- 172.21.15.46:0/2181468062 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e5000e6f0 con 0x7f4e58103440 2024-04-26T21:49:50.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.137+0000 7f4e4e7fc640 1 -- 172.21.15.46:0/2181468062 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e5000ed40 con 0x7f4e58103440 2024-04-26T21:49:50.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.137+0000 7f4e5f7f0640 1 -- 172.21.15.46:0/2181468062 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e5806aff0 con 0x7f4e58103440 2024-04-26T21:49:50.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.137+0000 7f4e4e7fc640 1 -- 172.21.15.46:0/2181468062 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e5000ac40 con 0x7f4e58103440 2024-04-26T21:49:50.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.137+0000 7f4e5f7f0640 1 -- 172.21.15.46:0/2181468062 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e5806b4c0 con 0x7f4e58103440 2024-04-26T21:49:50.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.138+0000 7f4e5f7f0640 1 -- 172.21.15.46:0/2181468062 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e58069ba0 con 0x7f4e58103440 2024-04-26T21:49:50.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.139+0000 7f4e4e7fc640 1 -- 172.21.15.46:0/2181468062 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4e5000e890 con 0x7f4e58103440 2024-04-26T21:49:50.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.139+0000 7f4e4e7fc640 1 --2- 172.21.15.46:0/2181468062 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4e440a3c40 0x7f4e440a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:50.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.140+0000 7f4e5d58c640 1 --2- 172.21.15.46:0/2181468062 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4e440a3c40 0x7f4e440a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:50.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.140+0000 7f4e4e7fc640 1 -- 172.21.15.46:0/2181468062 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7f4e500d2230 con 0x7f4e58103440 2024-04-26T21:49:50.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.140+0000 7f4e5d58c640 1 --2- 172.21.15.46:0/2181468062 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4e440a3c40 0x7f4e440a6100 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4e581028e0 tx=0x7f4e54009670 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:50.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.143+0000 7f4e4e7fc640 1 -- 172.21.15.46:0/2181468062 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4e5009eae0 con 0x7f4e58103440 2024-04-26T21:49:50.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.330+0000 7f4e5f7f0640 1 -- 172.21.15.46:0/2181468062 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4e581086a0 con 0x7f4e58103440 2024-04-26T21:49:50.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.331+0000 7f4e4e7fc640 1 -- 172.21.15.46:0/2181468062 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22602 (secure 0 0 0) 0x7f4e5000ae20 con 0x7f4e58103440 2024-04-26T21:49:50.332 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-04-26T21:49:50.332 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":87,"fsid":"36c7013a-0413-11ef-bc93-c7b262605968","created":"2024-04-26T21:24:30.481156+0000","modified":"2024-04-26T21:33:31.051554+0000","last_up_change":"2024-04-26T21:31:50.100491+0000","last_in_change":"2024-04-26T21:31:29.643428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:27:57.935988+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:32:42.195211+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:32:43.162114+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f53c8702-e4da-435f-a9dc-3911d0d36ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6803","nonce":2762665125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6805","nonce":2762665125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6809","nonce":2762665125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2762665125},{"type":"v1","addr":"172.21.15.46:6807","nonce":2762665125}]},"public_addr":"172.21.15.46:6803/2762665125","cluster_addr":"172.21.15.46:6805/2762665125","heartbeat_back_addr":"172.21.15.46:6809/2762665125","heartbeat_front_addr":"172.21.15.46:6807/2762665125","state":["exists","up"]},{"osd":1,"uuid":"fc4d6f52-fc8a-4efd-80c9-c092b57e7f42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6801","nonce":1628008970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6803","nonce":1628008970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6807","nonce":1628008970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1628008970},{"type":"v1","addr":"172.21.15.50:6805","nonce":1628008970}]},"public_addr":"172.21.15.50:6801/1628008970","cluster_addr":"172.21.15.50:6803/1628008970","heartbeat_back_addr":"172.21.15.50:6807/1628008970","heartbeat_front_addr":"172.21.15.50:6805/1628008970","state":["exists","up"]},{"osd":2,"uuid":"26d1baa1-4ea1-4b7e-9fcf-4f8bbf7329c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6800","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6801","nonce":1513279137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6802","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6803","nonce":1513279137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6806","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6807","nonce":1513279137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6804","nonce":1513279137},{"type":"v1","addr":"172.21.15.142:6805","nonce":1513279137}]},"public_addr":"172.21.15.142:6801/1513279137","cluster_addr":"172.21.15.142:6803/1513279137","heartbeat_back_addr":"172.21.15.142:6807/1513279137","heartbeat_front_addr":"172.21.15.142:6805/1513279137","state":["exists","up"]},{"osd":3,"uuid":"34d2401f-5fcc-491c-8fcf-c3c492611f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6811","nonce":2435181760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6813","nonce":2435181760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6817","nonce":2435181760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2435181760},{"type":"v1","addr":"172.21.15.46:6815","nonce":2435181760}]},"public_addr":"172.21.15.46:6811/2435181760","cluster_addr":"172.21.15.46:6813/2435181760","heartbeat_back_addr":"172.21.15.46:6817/2435181760","heartbeat_front_addr":"172.21.15.46:6815/2435181760","state":["exists","up"]},{"osd":4,"uuid":"986bb709-e4cf-400d-982e-c3f5ac413e0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6809","nonce":3177333250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6811","nonce":3177333250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6815","nonce":3177333250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3177333250},{"type":"v1","addr":"172.21.15.50:6813","nonce":3177333250}]},"public_addr":"172.21.15.50:6809/3177333250","cluster_addr":"172.21.15.50:6811/3177333250","heartbeat_back_addr":"172.21.15.50:6815/3177333250","heartbeat_front_addr":"172.21.15.50:6813/3177333250","state":["exists","up"]},{"osd":5,"uuid":"b7d65851-ebe1-45e2-833d-8022446f2698","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6808","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6809","nonce":4113107737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6810","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6811","nonce":4113107737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6814","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6815","nonce":4113107737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6812","nonce":4113107737},{"type":"v1","addr":"172.21.15.142:6813","nonce":4113107737}]},"public_addr":"172.21.15.142:6809/4113107737","cluster_addr":"172.21.15.142:6811/4113107737","heartbeat_back_addr":"172.21.15.142:6815/4113107737","heartbeat_front_addr":"172.21.15.142:6813/4113107737","state":["exists","up"]},{"osd":6,"uuid":"2a3e96d7-d5bf-44ff-9134-29f1420c0e4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6819","nonce":2325162882}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6821","nonce":2325162882}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6825","nonce":2325162882}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2325162882},{"type":"v1","addr":"172.21.15.46:6823","nonce":2325162882}]},"public_addr":"172.21.15.46:6819/2325162882","cluster_addr":"172.21.15.46:6821/2325162882","heartbeat_back_addr":"172.21.15.46:6825/2325162882","heartbeat_front_addr":"172.21.15.46:6823/2325162882","state":["exists","up"]},{"osd":7,"uuid":"e53ae885-23b6-46fd-b4b1-c62f31a6921f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6817","nonce":820310228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6819","nonce":820310228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6823","nonce":820310228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":820310228},{"type":"v1","addr":"172.21.15.50:6821","nonce":820310228}]},"public_addr":"172.21.15.50:6817/820310228","cluster_addr":"172.21.15.50:6819/820310228","heartbeat_back_addr":"172.21.15.50:6823/820310228","heartbeat_front_addr":"172.21.15.50:6821/820310228","state":["exists","up"]},{"osd":8,"uuid":"356e2692-16a9-4045-b58f-c8f9cb5d7241","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6816","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6817","nonce":1824918711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6818","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6819","nonce":1824918711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6822","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6823","nonce":1824918711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6820","nonce":1824918711},{"type":"v1","addr":"172.21.15.142:6821","nonce":1824918711}]},"public_addr":"172.21.15.142:6817/1824918711","cluster_addr":"172.21.15.142:6819/1824918711","heartbeat_back_addr":"172.21.15.142:6823/1824918711","heartbeat_front_addr":"172.21.15.142:6821/1824918711","state":["exists","up"]},{"osd":9,"uuid":"8e55570e-77e5-4892-8b9f-77dcd03a4eec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6827","nonce":3601947736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6829","nonce":3601947736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6833","nonce":3601947736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":3601947736},{"type":"v1","addr":"172.21.15.46:6831","nonce":3601947736}]},"public_addr":"172.21.15.46:6827/3601947736","cluster_addr":"172.21.15.46:6829/3601947736","heartbeat_back_addr":"172.21.15.46:6833/3601947736","heartbeat_front_addr":"172.21.15.46:6831/3601947736","state":["exists","up"]},{"osd":10,"uuid":"366389fa-fb9a-408b-b5c2-343c054c5a64","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6825","nonce":3135765006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6827","nonce":3135765006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6831","nonce":3135765006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":3135765006},{"type":"v1","addr":"172.21.15.50:6829","nonce":3135765006}]},"public_addr":"172.21.15.50:6825/3135765006","cluster_addr":"172.21.15.50:6827/3135765006","heartbeat_back_addr":"172.21.15.50:6831/3135765006","heartbeat_front_addr":"172.21.15.50:6829/3135765006","state":["exists","up"]},{"osd":11,"uuid":"7b6e3d93-59e1-4853-b4a2-261b728c47a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6824","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6825","nonce":1482712612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6826","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6827","nonce":1482712612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6830","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6831","nonce":1482712612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.142:6828","nonce":1482712612},{"type":"v1","addr":"172.21.15.142:6829","nonce":1482712612}]},"public_addr":"172.21.15.142:6825/1482712612","cluster_addr":"172.21.15.142:6827/1482712612","heartbeat_back_addr":"172.21.15.142:6831/1482712612","heartbeat_front_addr":"172.21.15.142:6829/1482712612","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:08.082015+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:32.634928+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:27:55.978864+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:21.796863+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:28:46.862518+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:11.440432+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:29:37.764777+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:03.374958+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:28.714052+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:30:55.761167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:22.330951+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:31:47.839334+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.27","mappings":[{"from":3,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/2465242088":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6801/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:6800/2053782984":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/1381997090":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/3507268217":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2570024356":"2024-04-27T21:25:02.078730+0000","172.21.15.46:0/2691061938":"2024-04-27T21:25:29.848343+0000","172.21.15.46:0/2775954801":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6800/2896098005":"2024-04-27T21:25:02.078730+0000","172.21.15.46:6801/2896098005":"2024-04-27T21:25:02.078730+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:49:50.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.333+0000 7f4e5f7f0640 1 -- 172.21.15.46:0/2181468062 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4e440a3c40 msgr2=0x7f4e440a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:50.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.333+0000 7f4e5f7f0640 1 --2- 172.21.15.46:0/2181468062 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4e440a3c40 0x7f4e440a6100 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4e581028e0 tx=0x7f4e54009670 comp rx=0 tx=0).stop 2024-04-26T21:49:50.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.333+0000 7f4e5f7f0640 1 -- 172.21.15.46:0/2181468062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e58103440 msgr2=0x7f4e5819f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:50.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.333+0000 7f4e5f7f0640 1 --2- 172.21.15.46:0/2181468062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e58103440 0x7f4e5819f010 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4e50007900 tx=0x7f4e50007dc0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.333+0000 7f4e5f7f0640 1 -- 172.21.15.46:0/2181468062 shutdown_connections 2024-04-26T21:49:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.333+0000 7f4e5f7f0640 1 --2- 172.21.15.46:0/2181468062 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f4e440a3c40 0x7f4e440a6100 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.333+0000 7f4e5f7f0640 1 --2- 172.21.15.46:0/2181468062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4e58103440 0x7f4e5819f010 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.333+0000 7f4e5f7f0640 1 --2- 172.21.15.46:0/2181468062 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f4e58102a80 0x7f4e58197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.333+0000 7f4e5f7f0640 1 --2- 172.21.15.46:0/2181468062 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4e58101890 0x7f4e58197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.333+0000 7f4e5f7f0640 1 -- 172.21.15.46:0/2181468062 >> 172.21.15.46:0/2181468062 conn(0x7f4e580fce20 msgr2=0x7f4e58105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.333+0000 7f4e5f7f0640 1 -- 172.21.15.46:0/2181468062 shutdown_connections 2024-04-26T21:49:50.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.333+0000 7f4e5f7f0640 1 -- 172.21.15.46:0/2181468062 wait complete. 2024-04-26T21:49:50.345 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-26T21:49:50.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.447+0000 7fe6de7c1640 1 -- 172.21.15.46:0/1139503308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6d8103450 msgr2=0x7fe6d8105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:50.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.447+0000 7fe6de7c1640 1 --2- 172.21.15.46:0/1139503308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6d8103450 0x7fe6d8105840 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8009900 tx=0x7fe6c802a410 comp rx=0 tx=0).stop 2024-04-26T21:49:50.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.447+0000 7fe6de7c1640 1 -- 172.21.15.46:0/1139503308 shutdown_connections 2024-04-26T21:49:50.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.447+0000 7fe6de7c1640 1 --2- 172.21.15.46:0/1139503308 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6d81086b0 0x7fe6d810aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.447+0000 7fe6de7c1640 1 --2- 172.21.15.46:0/1139503308 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6d8105d80 0x7fe6d8108170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.447+0000 7fe6de7c1640 1 --2- 172.21.15.46:0/1139503308 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6d8103450 0x7fe6d8105840 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.447+0000 7fe6de7c1640 1 -- 172.21.15.46:0/1139503308 >> 172.21.15.46:0/1139503308 conn(0x7fe6d80fcde0 msgr2=0x7fe6d80ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:50.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.447+0000 7fe6de7c1640 1 -- 172.21.15.46:0/1139503308 shutdown_connections 2024-04-26T21:49:50.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.447+0000 7fe6de7c1640 1 -- 172.21.15.46:0/1139503308 wait complete. 2024-04-26T21:49:50.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.448+0000 7fe6de7c1640 1 Processor -- start 2024-04-26T21:49:50.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.448+0000 7fe6de7c1640 1 -- start start 2024-04-26T21:49:50.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.448+0000 7fe6de7c1640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6d8103450 0x7fe6d819ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:50.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.448+0000 7fe6de7c1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6d8105d80 0x7fe6d819b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:50.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.448+0000 7fe6de7c1640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6d81086b0 0x7fe6d81a23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:50.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.448+0000 7fe6de7c1640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d810e490 con 0x7fe6d8105d80 2024-04-26T21:49:50.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.448+0000 7fe6de7c1640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d810e310 con 0x7fe6d81086b0 2024-04-26T21:49:50.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.448+0000 7fe6de7c1640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d810e610 con 0x7fe6d8103450 2024-04-26T21:49:50.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.448+0000 7fe6d77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6d8105d80 0x7fe6d819b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:50.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.448+0000 7fe6d77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6d8105d80 0x7fe6d819b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44860/0 (socket says 172.21.15.46:44860) 2024-04-26T21:49:50.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.448+0000 7fe6d77fe640 1 -- 172.21.15.46:0/3495229272 learned_addr learned my addr 172.21.15.46:0/3495229272 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:49:50.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.448+0000 7fe6d7fff640 1 --2- 172.21.15.46:0/3495229272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6d8103450 0x7fe6d819ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:50.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.449+0000 7fe6dcd5e640 1 --2- 172.21.15.46:0/3495229272 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6d81086b0 0x7fe6d81a23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:50.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.449+0000 7fe6d77fe640 1 -- 172.21.15.46:0/3495229272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6d8103450 msgr2=0x7fe6d819ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:50.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.449+0000 7fe6d77fe640 1 --2- 172.21.15.46:0/3495229272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6d8103450 0x7fe6d819ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.449+0000 7fe6d77fe640 1 -- 172.21.15.46:0/3495229272 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6d81086b0 msgr2=0x7fe6d81a23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:50.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.449+0000 7fe6d77fe640 1 --2- 172.21.15.46:0/3495229272 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6d81086b0 0x7fe6d81a23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.449+0000 7fe6d77fe640 1 -- 172.21.15.46:0/3495229272 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6d81a2a60 con 0x7fe6d8105d80 2024-04-26T21:49:50.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.449+0000 7fe6d77fe640 1 --2- 172.21.15.46:0/3495229272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6d8105d80 0x7fe6d819b380 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc00e9a0 tx=0x7fe6cc00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:50.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.449+0000 7fe6d57fa640 1 -- 172.21.15.46:0/3495229272 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc00cd80 con 0x7fe6d8105d80 2024-04-26T21:49:50.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.449+0000 7fe6d57fa640 1 -- 172.21.15.46:0/3495229272 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6cc0035f0 con 0x7fe6d8105d80 2024-04-26T21:49:50.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.449+0000 7fe6de7c1640 1 -- 172.21.15.46:0/3495229272 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6d81a2d50 con 0x7fe6d8105d80 2024-04-26T21:49:50.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.449+0000 7fe6d57fa640 1 -- 172.21.15.46:0/3495229272 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc010690 con 0x7fe6d8105d80 2024-04-26T21:49:50.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.449+0000 7fe6de7c1640 1 -- 172.21.15.46:0/3495229272 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6d81a3240 con 0x7fe6d8105d80 2024-04-26T21:49:50.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.450+0000 7fe6de7c1640 1 -- 172.21.15.46:0/3495229272 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6d810a1a0 con 0x7fe6d8105d80 2024-04-26T21:49:50.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.451+0000 7fe6d57fa640 1 -- 172.21.15.46:0/3495229272 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe6cc004000 con 0x7fe6d8105d80 2024-04-26T21:49:50.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.452+0000 7fe6d57fa640 1 --2- 172.21.15.46:0/3495229272 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe6c00a3c40 0x7fe6c00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:50.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.452+0000 7fe6d57fa640 1 -- 172.21.15.46:0/3495229272 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8190+0+0 (secure 0 0 0) 0x7fe6cc0d0940 con 0x7fe6d8105d80 2024-04-26T21:49:50.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.452+0000 7fe6d7fff640 1 --2- 172.21.15.46:0/3495229272 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe6c00a3c40 0x7fe6c00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:50.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.453+0000 7fe6d7fff640 1 --2- 172.21.15.46:0/3495229272 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe6c00a3c40 0x7fe6c00a6100 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8000c00 tx=0x7fe6c80314a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:50.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.456+0000 7fe6d57fa640 1 -- 172.21.15.46:0/3495229272 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe6cc0042f0 con 0x7fe6d8105d80 2024-04-26T21:49:50.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.683+0000 7fe6de7c1640 1 -- 172.21.15.46:0/3495229272 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fe6d802dd20 con 0x7fe6d8105d80 2024-04-26T21:49:50.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.685+0000 7fe6d57fa640 1 -- 172.21.15.46:0/3495229272 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v24) ==== 161+0+0 (secure 0 0 0) 0x7fe6cc09d280 con 0x7fe6d8105d80 2024-04-26T21:49:50.686 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-26T21:49:50.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.688+0000 7fe6de7c1640 1 -- 172.21.15.46:0/3495229272 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe6c00a3c40 msgr2=0x7fe6c00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:50.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.688+0000 7fe6de7c1640 1 --2- 172.21.15.46:0/3495229272 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe6c00a3c40 0x7fe6c00a6100 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8000c00 tx=0x7fe6c80314a0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.688+0000 7fe6de7c1640 1 -- 172.21.15.46:0/3495229272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6d8105d80 msgr2=0x7fe6d819b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:50.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.688+0000 7fe6de7c1640 1 --2- 172.21.15.46:0/3495229272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6d8105d80 0x7fe6d819b380 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc00e9a0 tx=0x7fe6cc00ee60 comp rx=0 tx=0).stop 2024-04-26T21:49:50.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.689+0000 7fe6de7c1640 1 -- 172.21.15.46:0/3495229272 shutdown_connections 2024-04-26T21:49:50.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.689+0000 7fe6de7c1640 1 --2- 172.21.15.46:0/3495229272 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fe6c00a3c40 0x7fe6c00a6100 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.689+0000 7fe6de7c1640 1 --2- 172.21.15.46:0/3495229272 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe6d81086b0 0x7fe6d81a23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.689+0000 7fe6de7c1640 1 --2- 172.21.15.46:0/3495229272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6d8105d80 0x7fe6d819b380 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.689+0000 7fe6de7c1640 1 --2- 172.21.15.46:0/3495229272 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fe6d8103450 0x7fe6d819ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.689+0000 7fe6de7c1640 1 -- 172.21.15.46:0/3495229272 >> 172.21.15.46:0/3495229272 conn(0x7fe6d80fcde0 msgr2=0x7fe6d8102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:50.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.689+0000 7fe6de7c1640 1 -- 172.21.15.46:0/3495229272 shutdown_connections 2024-04-26T21:49:50.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.689+0000 7fe6de7c1640 1 -- 172.21.15.46:0/3495229272 wait complete. 2024-04-26T21:49:50.701 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-26T21:49:50.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.805+0000 7fc800139640 1 -- 172.21.15.46:0/535012052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f8103b80 msgr2=0x7fc7f8104000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:50.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.806+0000 7fc800139640 1 --2- 172.21.15.46:0/535012052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f8103b80 0x7fc7f8104000 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec009900 tx=0x7fc7ec02a410 comp rx=0 tx=0).stop 2024-04-26T21:49:50.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.806+0000 7fc800139640 1 -- 172.21.15.46:0/535012052 shutdown_connections 2024-04-26T21:49:50.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.806+0000 7fc800139640 1 --2- 172.21.15.46:0/535012052 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7f8104540 0x7fc7f810add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.806+0000 7fc800139640 1 --2- 172.21.15.46:0/535012052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f8103b80 0x7fc7f8104000 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.806+0000 7fc800139640 1 --2- 172.21.15.46:0/535012052 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7f8102990 0x7fc7f8102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.806+0000 7fc800139640 1 -- 172.21.15.46:0/535012052 >> 172.21.15.46:0/535012052 conn(0x7fc7f80fdf00 msgr2=0x7fc7f8100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:50.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.806+0000 7fc800139640 1 -- 172.21.15.46:0/535012052 shutdown_connections 2024-04-26T21:49:50.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.806+0000 7fc800139640 1 -- 172.21.15.46:0/535012052 wait complete. 2024-04-26T21:49:50.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.806+0000 7fc800139640 1 Processor -- start 2024-04-26T21:49:50.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.806+0000 7fc800139640 1 -- start start 2024-04-26T21:49:50.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.806+0000 7fc800139640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7f8102990 0x7fc7f819ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:50.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.807+0000 7fc800139640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f8103b80 0x7fc7f819b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:50.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.807+0000 7fc800139640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7f8104540 0x7fc7f81a2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:50.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.807+0000 7fc800139640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f810e4c0 con 0x7fc7f8103b80 2024-04-26T21:49:50.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.807+0000 7fc800139640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f810e340 con 0x7fc7f8104540 2024-04-26T21:49:50.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.807+0000 7fc800139640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f810e640 con 0x7fc7f8102990 2024-04-26T21:49:50.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.807+0000 7fc7fd6d4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f8103b80 0x7fc7f819b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:50.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.807+0000 7fc7fd6d4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f8103b80 0x7fc7f819b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44884/0 (socket says 172.21.15.46:44884) 2024-04-26T21:49:50.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.807+0000 7fc7fd6d4640 1 -- 172.21.15.46:0/850519369 learned_addr learned my addr 172.21.15.46:0/850519369 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:49:50.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.807+0000 7fc7fded5640 1 --2- 172.21.15.46:0/850519369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7f8102990 0x7fc7f819ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:50.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.807+0000 7fc7fe6d6640 1 --2- 172.21.15.46:0/850519369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7f8104540 0x7fc7f81a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:50.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.807+0000 7fc7fd6d4640 1 -- 172.21.15.46:0/850519369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7f8102990 msgr2=0x7fc7f819ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:50.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.807+0000 7fc7fd6d4640 1 --2- 172.21.15.46:0/850519369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7f8102990 0x7fc7f819ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.807+0000 7fc7fd6d4640 1 -- 172.21.15.46:0/850519369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7f8104540 msgr2=0x7fc7f81a2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:50.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.807+0000 7fc7fd6d4640 1 --2- 172.21.15.46:0/850519369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7f8104540 0x7fc7f81a2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:50.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.807+0000 7fc7fd6d4640 1 -- 172.21.15.46:0/850519369 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7f81a2a00 con 0x7fc7f8103b80 2024-04-26T21:49:50.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.807+0000 7fc7fd6d4640 1 --2- 172.21.15.46:0/850519369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f8103b80 0x7fc7f819b320 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec005940 tx=0x7fc7ec0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:50.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.807+0000 7fc7f6ffd640 1 -- 172.21.15.46:0/850519369 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ec03f030 con 0x7fc7f8103b80 2024-04-26T21:49:50.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.807+0000 7fc7f6ffd640 1 -- 172.21.15.46:0/850519369 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7ec003cb0 con 0x7fc7f8103b80 2024-04-26T21:49:50.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.807+0000 7fc800139640 1 -- 172.21.15.46:0/850519369 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7f81a2c90 con 0x7fc7f8103b80 2024-04-26T21:49:50.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.807+0000 7fc7f6ffd640 1 -- 172.21.15.46:0/850519369 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ec035070 con 0x7fc7f8103b80 2024-04-26T21:49:50.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.807+0000 7fc800139640 1 -- 172.21.15.46:0/850519369 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7f81a3180 con 0x7fc7f8103b80 2024-04-26T21:49:50.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.808+0000 7fc800139640 1 -- 172.21.15.46:0/850519369 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7f81a3350 con 0x7fc7f8103b80 2024-04-26T21:49:50.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.809+0000 7fc7f6ffd640 1 -- 172.21.15.46:0/850519369 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc7ec042030 con 0x7fc7f8103b80 2024-04-26T21:49:50.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.810+0000 7fc7f6ffd640 1 --2- 172.21.15.46:0/850519369 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fc7e00a3c40 0x7fc7e00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:50.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.810+0000 7fc7fded5640 1 --2- 172.21.15.46:0/850519369 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fc7e00a3c40 0x7fc7e00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:50.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.810+0000 7fc7f6ffd640 1 -- 172.21.15.46:0/850519369 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8448+0+0 (secure 0 0 0) 0x7fc7ec0f0ac0 con 0x7fc7f8103b80 2024-04-26T21:49:50.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.810+0000 7fc7fded5640 1 --2- 172.21.15.46:0/850519369 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fc7e00a3c40 0x7fc7e00a6100 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc7f81039e0 tx=0x7fc7e8009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:50.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:50.815+0000 7fc7f6ffd640 1 -- 172.21.15.46:0/850519369 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc7ec0bd270 con 0x7fc7f8103b80 2024-04-26T21:49:51.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:50 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/1290160843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:49:51.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:50 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/2181468062' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:49:51.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:50 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3495229272' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T21:49:51.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:50 smithi142 ceph-mon[32944]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T21:49:51.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:50 smithi142 ceph-mon[32944]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T21:49:51.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:50 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/3495229272' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T21:49:51.042 INFO:journalctl@ceph.mds.c.smithi142.stdout:Apr 26 21:49:50 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-c[63877]: ignoring --setuser ceph since I am not root 2024-04-26T21:49:51.042 INFO:journalctl@ceph.mds.c.smithi142.stdout:Apr 26 21:49:50 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-c[63877]: ignoring --setgroup ceph since I am not root 2024-04-26T21:49:51.042 INFO:journalctl@ceph.mds.c.smithi142.stdout:Apr 26 21:49:50 smithi142 ceph-mds[63881]: -- 172.21.15.142:0/2454004376 <== mon.0 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e2dec17a40 con 0x55e2debeb000 2024-04-26T21:49:51.042 INFO:journalctl@ceph.mds.c.smithi142.stdout:Apr 26 21:49:50 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-c[63877]: starting mds.c at 2024-04-26T21:49:51.042 INFO:journalctl@ceph.mds.f.smithi142.stdout:Apr 26 21:49:50 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-f[63568]: ignoring --setuser ceph since I am not root 2024-04-26T21:49:51.043 INFO:journalctl@ceph.mds.f.smithi142.stdout:Apr 26 21:49:50 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-f[63568]: ignoring --setgroup ceph since I am not root 2024-04-26T21:49:51.043 INFO:journalctl@ceph.mds.f.smithi142.stdout:Apr 26 21:49:50 smithi142 ceph-mds[63572]: -- 172.21.15.142:0/1532773518 <== mon.1 v2:172.21.15.142:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562eb02a1a40 con 0x562eb0274800 2024-04-26T21:49:51.043 INFO:journalctl@ceph.mds.f.smithi142.stdout:Apr 26 21:49:50 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-f[63568]: starting mds.f at 2024-04-26T21:49:51.043 INFO:journalctl@ceph.mds.i.smithi142.stdout:Apr 26 21:49:50 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-i[64184]: ignoring --setuser ceph since I am not root 2024-04-26T21:49:51.043 INFO:journalctl@ceph.mds.i.smithi142.stdout:Apr 26 21:49:50 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-i[64184]: ignoring --setgroup ceph since I am not root 2024-04-26T21:49:51.044 INFO:journalctl@ceph.mds.i.smithi142.stdout:Apr 26 21:49:50 smithi142 ceph-mds[64188]: -- 172.21.15.142:0/4251985101 <== mon.1 v2:172.21.15.142:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55579a46da40 con 0x55579a440c00 2024-04-26T21:49:51.044 INFO:journalctl@ceph.mds.i.smithi142.stdout:Apr 26 21:49:50 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-i[64184]: starting mds.i at 2024-04-26T21:49:51.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.041+0000 7fc800139640 1 -- 172.21.15.46:0/850519369 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fc7f81a36e0 con 0x7fc7f8103b80 2024-04-26T21:49:51.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:50 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mon-a[29197]: 2024-04-26T21:49:50.684+0000 7fe0d6096700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T21:49:51.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:50 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/1290160843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:49:51.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:50 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2181468062' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:49:51.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:50 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3495229272' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T21:49:51.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:50 smithi046 ceph-mon[29201]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T21:49:51.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:50 smithi046 ceph-mon[29201]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T21:49:51.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:50 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/3495229272' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T21:49:51.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:50 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/1290160843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:49:51.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:50 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/2181468062' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:49:51.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:50 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3495229272' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T21:49:51.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:50 smithi050 ceph-mon[32753]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T21:49:51.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:50 smithi050 ceph-mon[32753]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T21:49:51.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:50 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/3495229272' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T21:49:51.119 INFO:journalctl@ceph.mds.b.smithi050.stdout:Apr 26 21:49:50 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-b[63049]: ignoring --setuser ceph since I am not root 2024-04-26T21:49:51.119 INFO:journalctl@ceph.mds.b.smithi050.stdout:Apr 26 21:49:50 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-b[63049]: ignoring --setgroup ceph since I am not root 2024-04-26T21:49:51.119 INFO:journalctl@ceph.mds.b.smithi050.stdout:Apr 26 21:49:50 smithi050 ceph-mds[63053]: -- 172.21.15.50:0/1853990980 <== mon.2 v2:172.21.15.50:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55acb4843a40 con 0x55acb4816c00 2024-04-26T21:49:51.119 INFO:journalctl@ceph.mds.b.smithi050.stdout:Apr 26 21:49:50 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-b[63049]: starting mds.b at 2024-04-26T21:49:51.119 INFO:journalctl@ceph.mds.h.smithi050.stdout:Apr 26 21:49:50 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-h[62739]: ignoring --setuser ceph since I am not root 2024-04-26T21:49:51.119 INFO:journalctl@ceph.mds.h.smithi050.stdout:Apr 26 21:49:50 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-h[62739]: ignoring --setgroup ceph since I am not root 2024-04-26T21:49:51.120 INFO:journalctl@ceph.mds.h.smithi050.stdout:Apr 26 21:49:50 smithi050 ceph-mds[62743]: -- 172.21.15.50:0/21732493 <== mon.1 v2:172.21.15.142:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56334faf9a40 con 0x56334facc800 2024-04-26T21:49:51.120 INFO:journalctl@ceph.mds.h.smithi050.stdout:Apr 26 21:49:50 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-h[62739]: starting mds.h at 2024-04-26T21:49:51.120 INFO:journalctl@ceph.mds.e.smithi050.stdout:Apr 26 21:49:50 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-e[62428]: ignoring --setuser ceph since I am not root 2024-04-26T21:49:51.120 INFO:journalctl@ceph.mds.e.smithi050.stdout:Apr 26 21:49:50 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-e[62428]: ignoring --setgroup ceph since I am not root 2024-04-26T21:49:51.120 INFO:journalctl@ceph.mds.e.smithi050.stdout:Apr 26 21:49:50 smithi050 ceph-mds[62432]: -- 172.21.15.50:0/2126924580 <== mon.0 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5571b7421a40 con 0x5571b73f5000 2024-04-26T21:49:51.120 INFO:journalctl@ceph.mds.e.smithi050.stdout:Apr 26 21:49:50 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-e[62428]: starting mds.e at 2024-04-26T21:49:51.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.706+0000 7fc7f6ffd640 1 -- 172.21.15.46:0/850519369 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v25) ==== 104+0+0 (secure 0 0 0) 0x7fc7ec003e60 con 0x7fc7f8103b80 2024-04-26T21:49:51.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.709+0000 7fc800139640 1 -- 172.21.15.46:0/850519369 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fc7e00a3c40 msgr2=0x7fc7e00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:51.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.709+0000 7fc800139640 1 --2- 172.21.15.46:0/850519369 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fc7e00a3c40 0x7fc7e00a6100 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc7f81039e0 tx=0x7fc7e8009500 comp rx=0 tx=0).stop 2024-04-26T21:49:51.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.709+0000 7fc800139640 1 -- 172.21.15.46:0/850519369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f8103b80 msgr2=0x7fc7f819b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:51.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.709+0000 7fc800139640 1 --2- 172.21.15.46:0/850519369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f8103b80 0x7fc7f819b320 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec005940 tx=0x7fc7ec0042f0 comp rx=0 tx=0).stop 2024-04-26T21:49:51.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.709+0000 7fc800139640 1 -- 172.21.15.46:0/850519369 shutdown_connections 2024-04-26T21:49:51.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.709+0000 7fc800139640 1 --2- 172.21.15.46:0/850519369 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7fc7e00a3c40 0x7fc7e00a6100 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:51.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.709+0000 7fc800139640 1 --2- 172.21.15.46:0/850519369 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc7f8104540 0x7fc7f81a2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:51.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.709+0000 7fc800139640 1 --2- 172.21.15.46:0/850519369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7f8103b80 0x7fc7f819b320 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:51.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.709+0000 7fc800139640 1 --2- 172.21.15.46:0/850519369 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7fc7f8102990 0x7fc7f819ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:51.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.709+0000 7fc800139640 1 -- 172.21.15.46:0/850519369 >> 172.21.15.46:0/850519369 conn(0x7fc7f80fdf00 msgr2=0x7fc7f8074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:51.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.709+0000 7fc800139640 1 -- 172.21.15.46:0/850519369 shutdown_connections 2024-04-26T21:49:51.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.709+0000 7fc800139640 1 -- 172.21.15.46:0/850519369 wait complete. 2024-04-26T21:49:51.725 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-26T21:49:51.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.838+0000 7f0d67124640 1 -- 172.21.15.46:0/3293102869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d60104530 msgr2=0x7f0d6010adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:51.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.838+0000 7f0d67124640 1 --2- 172.21.15.46:0/3293102869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d60104530 0x7f0d6010adc0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f0d58009ca0 tx=0x7f0d5802a8c0 comp rx=0 tx=0).stop 2024-04-26T21:49:51.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.838+0000 7f0d67124640 1 -- 172.21.15.46:0/3293102869 shutdown_connections 2024-04-26T21:49:51.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.838+0000 7f0d67124640 1 --2- 172.21.15.46:0/3293102869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d60104530 0x7f0d6010adc0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:51.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.838+0000 7f0d67124640 1 --2- 172.21.15.46:0/3293102869 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d60103b70 0x7f0d60103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:51.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.838+0000 7f0d67124640 1 --2- 172.21.15.46:0/3293102869 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d60102980 0x7f0d60102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:51.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.838+0000 7f0d67124640 1 -- 172.21.15.46:0/3293102869 >> 172.21.15.46:0/3293102869 conn(0x7f0d600fdf10 msgr2=0x7f0d60100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:51.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.838+0000 7f0d67124640 1 -- 172.21.15.46:0/3293102869 shutdown_connections 2024-04-26T21:49:51.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.838+0000 7f0d67124640 1 -- 172.21.15.46:0/3293102869 wait complete. 2024-04-26T21:49:51.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.838+0000 7f0d67124640 1 Processor -- start 2024-04-26T21:49:51.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.838+0000 7f0d67124640 1 -- start start 2024-04-26T21:49:51.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.839+0000 7f0d67124640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d60102980 0x7f0d600766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:51.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.839+0000 7f0d67124640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d60103b70 0x7f0d60074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:51.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.839+0000 7f0d67124640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d60104530 0x7f0d60075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:51.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.839+0000 7f0d67124640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7f0d6010e570 con 0x7f0d60103b70 2024-04-26T21:49:51.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.839+0000 7f0d67124640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0d6010e3f0 con 0x7f0d60102980 2024-04-26T21:49:51.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.839+0000 7f0d67124640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7f0d6010e6f0 con 0x7f0d60104530 2024-04-26T21:49:51.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.839+0000 7f0d5ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d60103b70 0x7f0d60074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:51.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.839+0000 7f0d5ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d60103b70 0x7f0d60074d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44902/0 (socket says 172.21.15.46:44902) 2024-04-26T21:49:51.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.839+0000 7f0d5ffff640 1 -- 172.21.15.46:0/85276652 learned_addr learned my addr 172.21.15.46:0/85276652 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:49:51.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.839+0000 7f0d64ec0640 1 --2- 172.21.15.46:0/85276652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d60102980 0x7f0d600766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:51.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.839+0000 7f0d656c1640 1 --2- 172.21.15.46:0/85276652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d60104530 0x7f0d60075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:51.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.839+0000 7f0d5ffff640 1 -- 172.21.15.46:0/85276652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d60104530 msgr2=0x7f0d60075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:51.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.839+0000 7f0d5ffff640 1 --2- 172.21.15.46:0/85276652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d60104530 0x7f0d60075260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:51.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.839+0000 7f0d5ffff640 1 -- 172.21.15.46:0/85276652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d60102980 msgr2=0x7f0d600766e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:51.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.839+0000 7f0d5ffff640 1 --2- 172.21.15.46:0/85276652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d60102980 0x7f0d600766e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:51.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.839+0000 7f0d5ffff640 1 -- 172.21.15.46:0/85276652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d60075af0 con 0x7f0d60103b70 2024-04-26T21:49:51.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.839+0000 7f0d656c1640 1 --2- 172.21.15.46:0/85276652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d60104530 0x7f0d60075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:49:51.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.839+0000 7f0d5ffff640 1 --2- 172.21.15.46:0/85276652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d60103b70 0x7f0d60074d20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0d5400c890 tx=0x7f0d5400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:51.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.839+0000 7f0d5dffb640 1 -- 172.21.15.46:0/85276652 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d54004220 con 0x7f0d60103b70 2024-04-26T21:49:51.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.839+0000 7f0d5dffb640 1 -- 172.21.15.46:0/85276652 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d540043c0 con 0x7f0d60103b70 2024-04-26T21:49:51.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.839+0000 7f0d67124640 1 -- 172.21.15.46:0/85276652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d601a2d30 con 0x7f0d60103b70 2024-04-26T21:49:51.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.839+0000 7f0d5dffb640 1 -- 172.21.15.46:0/85276652 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d54004a00 con 0x7f0d60103b70 2024-04-26T21:49:51.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.840+0000 7f0d67124640 1 -- 172.21.15.46:0/85276652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d601a3290 con 0x7f0d60103b70 2024-04-26T21:49:51.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.841+0000 7f0d67124640 1 -- 172.21.15.46:0/85276652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d60109ca0 con 0x7f0d60103b70 2024-04-26T21:49:51.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.841+0000 7f0d5dffb640 1 -- 172.21.15.46:0/85276652 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0d540079b0 con 0x7f0d60103b70 2024-04-26T21:49:51.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.842+0000 7f0d5dffb640 1 --2- 172.21.15.46:0/85276652 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f0d480a3cc0 0x7f0d480a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:51.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.842+0000 7f0d5dffb640 1 -- 172.21.15.46:0/85276652 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8448+0+0 (secure 0 0 0) 0x7f0d540d0420 con 0x7f0d60103b70 2024-04-26T21:49:51.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.842+0000 7f0d64ec0640 1 --2- 172.21.15.46:0/85276652 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f0d480a3cc0 0x7f0d480a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:51.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.843+0000 7f0d64ec0640 1 --2- 172.21.15.46:0/85276652 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f0d480a3cc0 0x7f0d480a6180 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0d601039d0 tx=0x7f0d50008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:51.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:51.847+0000 7f0d5dffb640 1 -- 172.21.15.46:0/85276652 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0d54014030 con 0x7f0d60103b70 2024-04-26T21:49:52.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.033+0000 7f0d67124640 1 -- 172.21.15.46:0/85276652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0d60063530 con 0x7f0d60103b70 2024-04-26T21:49:52.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:51 smithi142 ceph-mon[32944]: pgmap v807: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-26T21:49:52.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:51 smithi142 ceph-mon[32944]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T21:49:52.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:51 smithi142 ceph-mon[32944]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-26T21:49:52.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:51 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/850519369' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T21:49:52.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:51 smithi046 ceph-mon[29201]: pgmap v807: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-26T21:49:52.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:51 smithi046 ceph-mon[29201]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T21:49:52.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:51 smithi046 ceph-mon[29201]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-26T21:49:52.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:51 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/850519369' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T21:49:52.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:51 smithi050 ceph-mon[32753]: pgmap v807: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-04-26T21:49:52.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:51 smithi050 ceph-mon[32753]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T21:49:52.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:51 smithi050 ceph-mon[32753]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-26T21:49:52.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:51 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/850519369' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T21:49:52.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.710+0000 7f0d5dffb640 1 -- 172.21.15.46:0/85276652 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v89) ==== 181+0+0 (secure 0 0 0) 0x7f0d5409cbd0 con 0x7f0d60103b70 2024-04-26T21:49:52.711 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2024-04-26T21:49:52.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.714+0000 7f0d67124640 1 -- 172.21.15.46:0/85276652 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f0d480a3cc0 msgr2=0x7f0d480a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:52.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.714+0000 7f0d67124640 1 --2- 172.21.15.46:0/85276652 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f0d480a3cc0 0x7f0d480a6180 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0d601039d0 tx=0x7f0d50008040 comp rx=0 tx=0).stop 2024-04-26T21:49:52.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.714+0000 7f0d67124640 1 -- 172.21.15.46:0/85276652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d60103b70 msgr2=0x7f0d60074d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:52.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.714+0000 7f0d67124640 1 --2- 172.21.15.46:0/85276652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d60103b70 0x7f0d60074d20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0d5400c890 tx=0x7f0d5400cd50 comp rx=0 tx=0).stop 2024-04-26T21:49:52.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.714+0000 7f0d67124640 1 -- 172.21.15.46:0/85276652 shutdown_connections 2024-04-26T21:49:52.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.714+0000 7f0d67124640 1 --2- 172.21.15.46:0/85276652 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7f0d480a3cc0 0x7f0d480a6180 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:52.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.714+0000 7f0d67124640 1 --2- 172.21.15.46:0/85276652 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7f0d60104530 0x7f0d60075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:52.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.714+0000 7f0d67124640 1 --2- 172.21.15.46:0/85276652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d60103b70 0x7f0d60074d20 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:52.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.715+0000 7f0d67124640 1 --2- 172.21.15.46:0/85276652 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d60102980 0x7f0d600766e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:52.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.715+0000 7f0d67124640 1 -- 172.21.15.46:0/85276652 >> 172.21.15.46:0/85276652 conn(0x7f0d600fdf10 msgr2=0x7f0d60106e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:52.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.715+0000 7f0d67124640 1 -- 172.21.15.46:0/85276652 shutdown_connections 2024-04-26T21:49:52.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.715+0000 7f0d67124640 1 -- 172.21.15.46:0/85276652 wait complete. 2024-04-26T21:49:52.729 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-26T21:49:52.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.841+0000 7ffa38551640 1 -- 172.21.15.46:0/784199102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa30102990 msgr2=0x7ffa30102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:52.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.841+0000 7ffa38551640 1 --2- 172.21.15.46:0/784199102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa30102990 0x7ffa30102db0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ffa20009ca0 tx=0x7ffa2002a8c0 comp rx=0 tx=0).stop 2024-04-26T21:49:52.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.841+0000 7ffa38551640 1 -- 172.21.15.46:0/784199102 shutdown_connections 2024-04-26T21:49:52.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.841+0000 7ffa38551640 1 --2- 172.21.15.46:0/784199102 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa30104540 0x7ffa3010add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:52.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.841+0000 7ffa38551640 1 --2- 172.21.15.46:0/784199102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa30103b80 0x7ffa30104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:52.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.841+0000 7ffa38551640 1 --2- 172.21.15.46:0/784199102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa30102990 0x7ffa30102db0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:52.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.841+0000 7ffa38551640 1 -- 172.21.15.46:0/784199102 >> 172.21.15.46:0/784199102 conn(0x7ffa300fdf20 msgr2=0x7ffa30100340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:52.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.841+0000 7ffa38551640 1 -- 172.21.15.46:0/784199102 shutdown_connections 2024-04-26T21:49:52.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.841+0000 7ffa38551640 1 -- 172.21.15.46:0/784199102 wait complete. 2024-04-26T21:49:52.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.842+0000 7ffa38551640 1 Processor -- start 2024-04-26T21:49:52.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.842+0000 7ffa38551640 1 -- start start 2024-04-26T21:49:52.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.842+0000 7ffa38551640 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa30102990 0x7ffa30198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:52.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.842+0000 7ffa38551640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa30103b80 0x7ffa301990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:52.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.842+0000 7ffa38551640 1 --2- >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa30104540 0x7ffa301a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:52.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.842+0000 7ffa38551640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3010e510 con 0x7ffa30103b80 2024-04-26T21:49:52.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.842+0000 7ffa38551640 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3010e390 con 0x7ffa30102990 2024-04-26T21:49:52.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.842+0000 7ffa38551640 1 -- --> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3010e690 con 0x7ffa30104540 2024-04-26T21:49:52.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.842+0000 7ffa35aec640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa30103b80 0x7ffa301990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:52.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.842+0000 7ffa35aec640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa30103b80 0x7ffa301990f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44930/0 (socket says 172.21.15.46:44930) 2024-04-26T21:49:52.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.842+0000 7ffa35aec640 1 -- 172.21.15.46:0/2858983270 learned_addr learned my addr 172.21.15.46:0/2858983270 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-04-26T21:49:52.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.842+0000 7ffa362ed640 1 --2- 172.21.15.46:0/2858983270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa30102990 0x7ffa30198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:52.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.842+0000 7ffa36aee640 1 --2- 172.21.15.46:0/2858983270 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa30104540 0x7ffa301a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:52.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.842+0000 7ffa35aec640 1 -- 172.21.15.46:0/2858983270 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa30104540 msgr2=0x7ffa301a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:52.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.842+0000 7ffa35aec640 1 --2- 172.21.15.46:0/2858983270 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa30104540 0x7ffa301a0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:52.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.842+0000 7ffa35aec640 1 -- 172.21.15.46:0/2858983270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa30102990 msgr2=0x7ffa30198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:52.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.842+0000 7ffa35aec640 1 --2- 172.21.15.46:0/2858983270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa30102990 0x7ffa30198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:52.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.842+0000 7ffa35aec640 1 -- 172.21.15.46:0/2858983270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa301a07d0 con 0x7ffa30103b80 2024-04-26T21:49:52.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.842+0000 7ffa36aee640 1 --2- 172.21.15.46:0/2858983270 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa30104540 0x7ffa301a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:49:52.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.843+0000 7ffa35aec640 1 --2- 172.21.15.46:0/2858983270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa30103b80 0x7ffa301990f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ffa2400d950 tx=0x7ffa2400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:52.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.843+0000 7ffa2f7fe640 1 -- 172.21.15.46:0/2858983270 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa24017070 con 0x7ffa30103b80 2024-04-26T21:49:52.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.843+0000 7ffa38551640 1 -- 172.21.15.46:0/2858983270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa301a0ac0 con 0x7ffa30103b80 2024-04-26T21:49:52.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.843+0000 7ffa2f7fe640 1 -- 172.21.15.46:0/2858983270 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa24009c30 con 0x7ffa30103b80 2024-04-26T21:49:52.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.843+0000 7ffa2f7fe640 1 -- 172.21.15.46:0/2858983270 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2400f620 con 0x7ffa30103b80 2024-04-26T21:49:52.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.843+0000 7ffa38551640 1 -- 172.21.15.46:0/2858983270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa301a1010 con 0x7ffa30103b80 2024-04-26T21:49:52.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.844+0000 7ffa38551640 1 -- 172.21.15.46:0/2858983270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa30109d30 con 0x7ffa30103b80 2024-04-26T21:49:52.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.845+0000 7ffa2f7fe640 1 -- 172.21.15.46:0/2858983270 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ffa24010460 con 0x7ffa30103b80 2024-04-26T21:49:52.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.845+0000 7ffa2f7fe640 1 --2- 172.21.15.46:0/2858983270 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ffa180a3c40 0x7ffa180a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:52.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.846+0000 7ffa362ed640 1 --2- 172.21.15.46:0/2858983270 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ffa180a3c40 0x7ffa180a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:52.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.846+0000 7ffa2f7fe640 1 -- 172.21.15.46:0/2858983270 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 7923+0+0 (secure 0 0 0) 0x7ffa240d0100 con 0x7ffa30103b80 2024-04-26T21:49:52.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.846+0000 7ffa362ed640 1 --2- 172.21.15.46:0/2858983270 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ffa180a3c40 0x7ffa180a6100 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ffa20000f80 tx=0x7ffa20005f30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:52.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:52.850+0000 7ffa2f7fe640 1 -- 172.21.15.46:0/2858983270 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ffa2409ca60 con 0x7ffa30103b80 2024-04-26T21:49:53.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:53.037+0000 7ffa38551640 1 -- 172.21.15.46:0/2858983270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7ffa30063560 con 0x7ffa30103b80 2024-04-26T21:49:53.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:52 smithi142 ceph-mon[32944]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T21:49:53.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:52 smithi142 ceph-mon[32944]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T21:49:53.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:52 smithi142 ceph-mon[32944]: Cluster is now healthy 2024-04-26T21:49:53.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:52 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/850519369' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T21:49:53.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:52 smithi142 ceph-mon[32944]: mds.? [v2:172.21.15.142:6835/2978071558,v1:172.21.15.142:6837/2978071558] up:boot 2024-04-26T21:49:53.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:52 smithi142 ceph-mon[32944]: mds.? [v2:172.21.15.50:6834/3446934241,v1:172.21.15.50:6835/3446934241] up:boot 2024-04-26T21:49:53.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:52 smithi142 ceph-mon[32944]: mds.? [v2:172.21.15.142:6834/1325055300,v1:172.21.15.142:6836/1325055300] up:boot 2024-04-26T21:49:53.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:52 smithi142 ceph-mon[32944]: mds.? [v2:172.21.15.142:6838/4123917318,v1:172.21.15.142:6839/4123917318] up:boot 2024-04-26T21:49:53.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:52 smithi142 ceph-mon[32944]: mds.? [v2:172.21.15.50:6836/3017379578,v1:172.21.15.50:6838/3017379578] up:boot 2024-04-26T21:49:53.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:52 smithi142 ceph-mon[32944]: mds.? [v2:172.21.15.50:6837/3382111533,v1:172.21.15.50:6839/3382111533] up:boot 2024-04-26T21:49:53.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:52 smithi142 ceph-mon[32944]: fsmap 12 up:standby 2024-04-26T21:49:53.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:52 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T21:49:53.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:52 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T21:49:53.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:52 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T21:49:53.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:52 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T21:49:53.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:52 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T21:49:53.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:52 smithi142 ceph-mon[32944]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T21:49:53.042 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:52 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/85276652' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T21:49:53.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:52 smithi046 ceph-mon[29201]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T21:49:53.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:52 smithi046 ceph-mon[29201]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T21:49:53.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:52 smithi046 ceph-mon[29201]: Cluster is now healthy 2024-04-26T21:49:53.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:52 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/850519369' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T21:49:53.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:52 smithi046 ceph-mon[29201]: mds.? [v2:172.21.15.142:6835/2978071558,v1:172.21.15.142:6837/2978071558] up:boot 2024-04-26T21:49:53.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:52 smithi046 ceph-mon[29201]: mds.? [v2:172.21.15.50:6834/3446934241,v1:172.21.15.50:6835/3446934241] up:boot 2024-04-26T21:49:53.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:52 smithi046 ceph-mon[29201]: mds.? [v2:172.21.15.142:6834/1325055300,v1:172.21.15.142:6836/1325055300] up:boot 2024-04-26T21:49:53.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:52 smithi046 ceph-mon[29201]: mds.? [v2:172.21.15.142:6838/4123917318,v1:172.21.15.142:6839/4123917318] up:boot 2024-04-26T21:49:53.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:52 smithi046 ceph-mon[29201]: mds.? [v2:172.21.15.50:6836/3017379578,v1:172.21.15.50:6838/3017379578] up:boot 2024-04-26T21:49:53.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:52 smithi046 ceph-mon[29201]: mds.? [v2:172.21.15.50:6837/3382111533,v1:172.21.15.50:6839/3382111533] up:boot 2024-04-26T21:49:53.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:52 smithi046 ceph-mon[29201]: fsmap 12 up:standby 2024-04-26T21:49:53.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:52 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T21:49:53.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:52 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T21:49:53.055 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:52 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T21:49:53.055 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:52 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T21:49:53.055 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:52 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T21:49:53.055 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:52 smithi046 ceph-mon[29201]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T21:49:53.055 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:52 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/85276652' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T21:49:53.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:52 smithi050 ceph-mon[32753]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T21:49:53.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:52 smithi050 ceph-mon[32753]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T21:49:53.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:52 smithi050 ceph-mon[32753]: Cluster is now healthy 2024-04-26T21:49:53.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:52 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/850519369' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T21:49:53.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:52 smithi050 ceph-mon[32753]: mds.? [v2:172.21.15.142:6835/2978071558,v1:172.21.15.142:6837/2978071558] up:boot 2024-04-26T21:49:53.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:52 smithi050 ceph-mon[32753]: mds.? [v2:172.21.15.50:6834/3446934241,v1:172.21.15.50:6835/3446934241] up:boot 2024-04-26T21:49:53.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:52 smithi050 ceph-mon[32753]: mds.? [v2:172.21.15.142:6834/1325055300,v1:172.21.15.142:6836/1325055300] up:boot 2024-04-26T21:49:53.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:52 smithi050 ceph-mon[32753]: mds.? [v2:172.21.15.142:6838/4123917318,v1:172.21.15.142:6839/4123917318] up:boot 2024-04-26T21:49:53.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:52 smithi050 ceph-mon[32753]: mds.? [v2:172.21.15.50:6836/3017379578,v1:172.21.15.50:6838/3017379578] up:boot 2024-04-26T21:49:53.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:52 smithi050 ceph-mon[32753]: mds.? [v2:172.21.15.50:6837/3382111533,v1:172.21.15.50:6839/3382111533] up:boot 2024-04-26T21:49:53.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:52 smithi050 ceph-mon[32753]: fsmap 12 up:standby 2024-04-26T21:49:53.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:52 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T21:49:53.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:52 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T21:49:53.119 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:52 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T21:49:53.120 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:52 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T21:49:53.120 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:52 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T21:49:53.120 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:52 smithi050 ceph-mon[32753]: from='mgr.14150 172.21.15.46:0/493789492' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T21:49:53.120 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:52 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/85276652' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T21:49:53.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:53.712+0000 7ffa2f7fe640 1 -- 172.21.15.46:0/2858983270 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v90) ==== 169+0+0 (secure 0 0 0) 0x7ffa24009de0 con 0x7ffa30103b80 2024-04-26T21:49:53.713 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2024-04-26T21:49:53.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:53.716+0000 7ffa38551640 1 -- 172.21.15.46:0/2858983270 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ffa180a3c40 msgr2=0x7ffa180a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:53.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:53.716+0000 7ffa38551640 1 --2- 172.21.15.46:0/2858983270 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ffa180a3c40 0x7ffa180a6100 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ffa20000f80 tx=0x7ffa20005f30 comp rx=0 tx=0).stop 2024-04-26T21:49:53.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:53.716+0000 7ffa38551640 1 -- 172.21.15.46:0/2858983270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa30103b80 msgr2=0x7ffa301990f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:53.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:53.716+0000 7ffa38551640 1 --2- 172.21.15.46:0/2858983270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa30103b80 0x7ffa301990f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ffa2400d950 tx=0x7ffa2400de10 comp rx=0 tx=0).stop 2024-04-26T21:49:53.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:53.716+0000 7ffa38551640 1 -- 172.21.15.46:0/2858983270 shutdown_connections 2024-04-26T21:49:53.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:53.716+0000 7ffa38551640 1 --2- 172.21.15.46:0/2858983270 >> [v2:172.21.15.46:6800/4004175376,v1:172.21.15.46:6801/4004175376] conn(0x7ffa180a3c40 0x7ffa180a6100 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:53.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:53.716+0000 7ffa38551640 1 --2- 172.21.15.46:0/2858983270 >> [v2:172.21.15.142:3300/0,v1:172.21.15.142:6789/0] conn(0x7ffa30104540 0x7ffa301a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:53.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:53.716+0000 7ffa38551640 1 --2- 172.21.15.46:0/2858983270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa30103b80 0x7ffa301990f0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:53.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:53.716+0000 7ffa38551640 1 --2- 172.21.15.46:0/2858983270 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa30102990 0x7ffa30198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:53.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:53.716+0000 7ffa38551640 1 -- 172.21.15.46:0/2858983270 >> 172.21.15.46:0/2858983270 conn(0x7ffa300fdf20 msgr2=0x7ffa30106f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:53.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:53.716+0000 7ffa38551640 1 -- 172.21.15.46:0/2858983270 shutdown_connections 2024-04-26T21:49:53.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-04-26T21:49:53.716+0000 7ffa38551640 1 -- 172.21.15.46:0/2858983270 wait complete. 2024-04-26T21:49:53.731 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T21:49:53.757 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T21:49:53.793 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T21:49:53.825 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-26T21:49:53.825 DEBUG:teuthology.orchestra.run.smithi046:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T21:49:53.841 DEBUG:teuthology.orchestra.run.smithi050:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T21:49:53.860 DEBUG:teuthology.orchestra.run.smithi142:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T21:49:53.882 INFO:tasks.cephadm:Stopping all daemons... 2024-04-26T21:49:53.883 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-26T21:49:53.883 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@mon.a 2024-04-26T21:49:54.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:53 smithi142 ceph-mon[32944]: pgmap v809: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-26T21:49:54.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:53 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/85276652' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T21:49:54.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:53 smithi142 ceph-mon[32944]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T21:49:54.041 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:53 smithi142 ceph-mon[32944]: from='client.? 172.21.15.46:0/2858983270' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T21:49:54.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:53 smithi046 ceph-mon[29201]: pgmap v809: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-26T21:49:54.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:53 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/85276652' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T21:49:54.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:53 smithi046 ceph-mon[29201]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T21:49:54.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:53 smithi046 ceph-mon[29201]: from='client.? 172.21.15.46:0/2858983270' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T21:49:54.054 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:53 smithi046 systemd[1]: Stopping Ceph mon.a for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:49:54.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:53 smithi050 ceph-mon[32753]: pgmap v809: 97 pgs: 97 active+clean; 634 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2024-04-26T21:49:54.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:53 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/85276652' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T21:49:54.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:53 smithi050 ceph-mon[32753]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T21:49:54.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:53 smithi050 ceph-mon[32753]: from='client.? 172.21.15.46:0/2858983270' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T21:49:54.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:54 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mon-a[29197]: 2024-04-26T21:49:54.197+0000 7fe0db8a1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:49:54.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:54 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mon-a[29197]: 2024-04-26T21:49:54.197+0000 7fe0db8a1700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-26T21:49:54.553 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:54 smithi046 podman[84539]: 2024-04-26 21:49:54.2804083 +0000 UTC m=+0.209103739 container died fc596ea263bf65540b7a94fda1bb803cbcc1ec30cfa0880760818f6c09e7de32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-26T21:49:55.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:54 smithi046 podman[84539]: 2024-04-26 21:49:54.907486655 +0000 UTC m=+0.836182131 container cleanup fc596ea263bf65540b7a94fda1bb803cbcc1ec30cfa0880760818f6c09e7de32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-26T21:49:55.053 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:54 smithi046 bash[84539]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-mon-a 2024-04-26T21:49:55.384 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:55 smithi046 podman[84553]: 2024-04-26 21:49:55.134098274 +0000 UTC m=+0.851493476 container remove fc596ea263bf65540b7a94fda1bb803cbcc1ec30cfa0880760818f6c09e7de32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:49:55.384 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:55 smithi046 systemd[1]: ceph-36c7013a-0413-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-26T21:49:55.385 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:55 smithi046 systemd[1]: Stopped Ceph mon.a for 36c7013a-0413-11ef-bc93-c7b262605968. 2024-04-26T21:49:55.385 INFO:journalctl@ceph.mon.a.smithi046.stdout:Apr 26 21:49:55 smithi046 systemd[1]: ceph-36c7013a-0413-11ef-bc93-c7b262605968@mon.a.service: Consumed 20.521s CPU time. 2024-04-26T21:49:55.386 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mon.a.service' 2024-04-26T21:49:55.445 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:49:55.445 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-26T21:49:55.445 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-26T21:49:55.446 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@mon.b 2024-04-26T21:49:55.768 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:55 smithi050 systemd[1]: Stopping Ceph mon.b for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:49:56.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:55 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mon-b[32749]: 2024-04-26T21:49:55.767+0000 7f8cf1c3e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:49:56.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:55 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mon-b[32749]: 2024-04-26T21:49:55.767+0000 7f8cf1c3e700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-26T21:49:56.118 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:55 smithi050 podman[67332]: 2024-04-26 21:49:55.900717485 +0000 UTC m=+0.249683801 container died 7e5f775ded483569d34b8a00c3092be900247cc9ceeaf5ca339fcde440741090 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mon-b, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-26T21:49:56.578 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:56 smithi050 podman[67332]: 2024-04-26 21:49:56.576979071 +0000 UTC m=+0.925945391 container cleanup 7e5f775ded483569d34b8a00c3092be900247cc9ceeaf5ca339fcde440741090 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mon-b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424) 2024-04-26T21:49:56.579 INFO:journalctl@ceph.mon.b.smithi050.stdout:Apr 26 21:49:56 smithi050 bash[67332]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-mon-b 2024-04-26T21:49:57.023 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mon.b.service' 2024-04-26T21:49:57.067 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:49:57.068 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-26T21:49:57.068 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-26T21:49:57.068 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@mon.c 2024-04-26T21:49:57.403 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:57 smithi142 systemd[1]: Stopping Ceph mon.c for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:49:57.787 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:57 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mon-c[32932]: 2024-04-26T21:49:57.402+0000 7f9ad0bf1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:49:57.787 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:57 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mon-c[32932]: 2024-04-26T21:49:57.402+0000 7f9ad0bf1700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-26T21:49:57.787 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:57 smithi142 podman[68498]: 2024-04-26 21:49:57.51534224 +0000 UTC m=+0.244732658 container died 633ca5d086a4fa27456b42610513f8b032b94de62cbd4e482162476e48979f17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mon-c, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD) 2024-04-26T21:49:58.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:57 smithi142 podman[68498]: 2024-04-26 21:49:57.962247329 +0000 UTC m=+0.691637756 container cleanup 633ca5d086a4fa27456b42610513f8b032b94de62cbd4e482162476e48979f17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mon-c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-26T21:49:58.040 INFO:journalctl@ceph.mon.c.smithi142.stdout:Apr 26 21:49:57 smithi142 bash[68498]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-mon-c 2024-04-26T21:49:58.384 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mon.c.service' 2024-04-26T21:49:58.432 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:49:58.432 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-26T21:49:58.432 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-26T21:49:58.432 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@mgr.x 2024-04-26T21:49:58.771 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:49:58 smithi046 systemd[1]: Stopping Ceph mgr.x for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:49:59.053 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:49:58 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:49:58.769+0000 7f94dd591700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:49:59.053 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:49:58 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x[29431]: 2024-04-26T21:49:58.769+0000 7f94dd591700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-26T21:49:59.053 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:49:58 smithi046 podman[84662]: 2024-04-26 21:49:58.801196017 +0000 UTC m=+0.165918579 container died d91b1acc0a72de05b448711dc23b2433847aecdf36f738e3006a2ef23253bcd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True) 2024-04-26T21:49:59.329 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:49:59 smithi046 podman[84662]: 2024-04-26 21:49:59.329723721 +0000 UTC m=+0.694446293 container cleanup d91b1acc0a72de05b448711dc23b2433847aecdf36f738e3006a2ef23253bcd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-26T21:49:59.737 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:49:59 smithi046 bash[84662]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x 2024-04-26T21:49:59.737 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Apr 26 21:49:59 smithi046 podman[84676]: 2024-04-26 21:49:59.539619333 +0000 UTC m=+0.737502873 container remove d91b1acc0a72de05b448711dc23b2433847aecdf36f738e3006a2ef23253bcd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T21:49:59.753 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-26T21:49:59.791 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:49:59.791 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-26T21:49:59.791 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-26T21:49:59.791 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@mgr.y 2024-04-26T21:50:00.115 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:49:59 smithi050 systemd[1]: Stopping Ceph mgr.y for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:50:00.368 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:50:00 smithi050 podman[67457]: 2024-04-26 21:50:00.128902265 +0000 UTC m=+0.143506965 container died 9a30793b0e455a581f2249bd613f3489e6ef15893d371f4594131a0b871b5d3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-26T21:50:00.868 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:50:00 smithi050 podman[67457]: 2024-04-26 21:50:00.570592131 +0000 UTC m=+0.585196876 container cleanup 9a30793b0e455a581f2249bd613f3489e6ef15893d371f4594131a0b871b5d3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:50:00.868 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:50:00 smithi050 bash[67457]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y 2024-04-26T21:50:01.164 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:50:00 smithi050 podman[67472]: 2024-04-26 21:50:00.914328817 +0000 UTC m=+0.777172610 container remove 9a30793b0e455a581f2249bd613f3489e6ef15893d371f4594131a0b871b5d3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True) 2024-04-26T21:50:01.165 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:50:00 smithi050 systemd[1]: ceph-36c7013a-0413-11ef-bc93-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-04-26T21:50:01.165 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:50:01 smithi050 systemd[1]: ceph-36c7013a-0413-11ef-bc93-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-04-26T21:50:01.165 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:50:01 smithi050 systemd[1]: Stopped Ceph mgr.y for 36c7013a-0413-11ef-bc93-c7b262605968. 2024-04-26T21:50:01.165 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 21:50:01 smithi050 systemd[1]: ceph-36c7013a-0413-11ef-bc93-c7b262605968@mgr.y.service: Consumed 14.618s CPU time. 2024-04-26T21:50:01.166 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-26T21:50:01.202 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:50:01.202 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-26T21:50:01.202 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-26T21:50:01.203 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@mgr.z 2024-04-26T21:50:01.521 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:50:01 smithi142 systemd[1]: Stopping Ceph mgr.z for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:50:01.790 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:50:01 smithi142 podman[68626]: 2024-04-26 21:50:01.522007304 +0000 UTC m=+0.134160837 container died 7662cfb726a521b9d91a41256b386ed3cd4e06b42274b826c4af2b52b4f24a95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240424) 2024-04-26T21:50:02.157 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:50:01 smithi142 podman[68626]: 2024-04-26 21:50:01.901126148 +0000 UTC m=+0.513279685 container cleanup 7662cfb726a521b9d91a41256b386ed3cd4e06b42274b826c4af2b52b4f24a95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-26T21:50:02.157 INFO:journalctl@ceph.mgr.z.smithi142.stdout:Apr 26 21:50:01 smithi142 bash[68626]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-mgr-z 2024-04-26T21:50:02.313 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-26T21:50:02.347 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:50:02.347 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-26T21:50:02.347 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-26T21:50:02.347 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.0 2024-04-26T21:50:02.659 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 26 21:50:02 smithi046 systemd[1]: Stopping Ceph osd.0 for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:50:03.053 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 26 21:50:02 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-0[43892]: 2024-04-26T21:50:02.658+0000 7fa00ac8a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:50:03.053 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 26 21:50:02 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-0[43892]: 2024-04-26T21:50:02.658+0000 7fa00ac8a700 -1 osd.0 90 *** Got signal Terminated *** 2024-04-26T21:50:03.053 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 26 21:50:02 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-0[43892]: 2024-04-26T21:50:02.658+0000 7fa00ac8a700 -1 osd.0 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T21:50:08.469 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 26 21:50:08 smithi046 podman[84798]: 2024-04-26 21:50:08.057706991 +0000 UTC m=+5.518170696 container died 91c7c1816fbb662093f24f2ec90b0df93dcdcfdf09cb59b9bd6a5770c196e532 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-0, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:50:08.803 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 26 21:50:08 smithi046 podman[84798]: 2024-04-26 21:50:08.576109976 +0000 UTC m=+6.036573678 container cleanup 91c7c1816fbb662093f24f2ec90b0df93dcdcfdf09cb59b9bd6a5770c196e532 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-26T21:50:08.803 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 26 21:50:08 smithi046 bash[84798]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-0 2024-04-26T21:50:09.217 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 26 21:50:08 smithi046 podman[84815]: 2024-04-26 21:50:08.850615669 +0000 UTC m=+0.782766511 container remove 91c7c1816fbb662093f24f2ec90b0df93dcdcfdf09cb59b9bd6a5770c196e532 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T21:50:09.487 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 26 21:50:09 smithi046 podman[84910]: 2024-04-26 21:50:09.121000533 +0000 UTC m=+0.019177722 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:50:09.487 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 26 21:50:09 smithi046 podman[84910]: 2024-04-26 21:50:09.259954717 +0000 UTC m=+0.158131934 container create a04cd35a297c1bf1ab746e941de877e3fc1ac174cf9399902e9f5736eb1666d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-0-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:50:09.787 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 26 21:50:09 smithi046 podman[84910]: 2024-04-26 21:50:09.487910871 +0000 UTC m=+0.386088052 container init a04cd35a297c1bf1ab746e941de877e3fc1ac174cf9399902e9f5736eb1666d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-26T21:50:09.787 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 26 21:50:09 smithi046 podman[84910]: 2024-04-26 21:50:09.491299366 +0000 UTC m=+0.389476539 container start a04cd35a297c1bf1ab746e941de877e3fc1ac174cf9399902e9f5736eb1666d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2) 2024-04-26T21:50:09.787 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 26 21:50:09 smithi046 podman[84910]: 2024-04-26 21:50:09.571100372 +0000 UTC m=+0.469277547 container attach a04cd35a297c1bf1ab746e941de877e3fc1ac174cf9399902e9f5736eb1666d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD) 2024-04-26T21:50:10.053 INFO:journalctl@ceph.osd.0.smithi046.stdout:Apr 26 21:50:09 smithi046 podman[84910]: 2024-04-26 21:50:09.812606576 +0000 UTC m=+0.710783752 container died a04cd35a297c1bf1ab746e941de877e3fc1ac174cf9399902e9f5736eb1666d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-26T21:50:10.522 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.0.service' 2024-04-26T21:50:10.561 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:50:10.561 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-26T21:50:10.561 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-26T21:50:10.561 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.1 2024-04-26T21:50:10.868 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 21:50:10 smithi050 systemd[1]: Stopping Ceph osd.1 for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:50:11.368 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 21:50:10 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-1[37343]: 2024-04-26T21:50:10.883+0000 7f4f559b3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:50:11.368 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 21:50:10 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-1[37343]: 2024-04-26T21:50:10.883+0000 7f4f559b3700 -1 osd.1 90 *** Got signal Terminated *** 2024-04-26T21:50:11.368 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 21:50:10 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-1[37343]: 2024-04-26T21:50:10.883+0000 7f4f559b3700 -1 osd.1 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T21:50:16.618 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 21:50:16 smithi050 podman[67582]: 2024-04-26 21:50:16.299641074 +0000 UTC m=+5.537313905 container died 14a8d8b3fd1c4b9763ab3684e9c2648d7bb30ee1474889400fe0b4409be8fe76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-26T21:50:16.910 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 21:50:16 smithi050 podman[67582]: 2024-04-26 21:50:16.70912958 +0000 UTC m=+5.946802408 container cleanup 14a8d8b3fd1c4b9763ab3684e9c2648d7bb30ee1474889400fe0b4409be8fe76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-1, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-26T21:50:16.911 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 21:50:16 smithi050 bash[67582]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-1 2024-04-26T21:50:17.341 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 21:50:16 smithi050 podman[67599]: 2024-04-26 21:50:16.910965368 +0000 UTC m=+0.606860443 container remove 14a8d8b3fd1c4b9763ab3684e9c2648d7bb30ee1474889400fe0b4409be8fe76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:50:17.618 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 21:50:17 smithi050 podman[67697]: 2024-04-26 21:50:17.244664276 +0000 UTC m=+0.018484129 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:50:17.618 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 21:50:17 smithi050 podman[67697]: 2024-04-26 21:50:17.35358537 +0000 UTC m=+0.127405193 container create 6b20fe96126d0ab90ab57f15cd4040e3fc2b21174462b008cd89df9a0d736780 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-26T21:50:17.926 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 21:50:17 smithi050 podman[67697]: 2024-04-26 21:50:17.623046476 +0000 UTC m=+0.396866299 container init 6b20fe96126d0ab90ab57f15cd4040e3fc2b21174462b008cd89df9a0d736780 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-26T21:50:17.926 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 21:50:17 smithi050 podman[67697]: 2024-04-26 21:50:17.626465827 +0000 UTC m=+0.400285658 container start 6b20fe96126d0ab90ab57f15cd4040e3fc2b21174462b008cd89df9a0d736780 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T21:50:17.926 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 21:50:17 smithi050 podman[67697]: 2024-04-26 21:50:17.839891949 +0000 UTC m=+0.613711786 container attach 6b20fe96126d0ab90ab57f15cd4040e3fc2b21174462b008cd89df9a0d736780 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-26T21:50:18.284 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 21:50:17 smithi050 podman[67697]: 2024-04-26 21:50:17.927008549 +0000 UTC m=+0.700828366 container died 6b20fe96126d0ab90ab57f15cd4040e3fc2b21174462b008cd89df9a0d736780 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T21:50:18.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.1.service' 2024-04-26T21:50:18.639 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:50:18.640 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-26T21:50:18.640 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-26T21:50:18.640 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.2 2024-04-26T21:50:18.954 INFO:journalctl@ceph.osd.2.smithi142.stdout:Apr 26 21:50:18 smithi142 systemd[1]: Stopping Ceph osd.2 for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:50:19.290 INFO:journalctl@ceph.osd.2.smithi142.stdout:Apr 26 21:50:18 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-2[37956]: 2024-04-26T21:50:18.953+0000 7fd8d7167700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:50:19.291 INFO:journalctl@ceph.osd.2.smithi142.stdout:Apr 26 21:50:18 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-2[37956]: 2024-04-26T21:50:18.953+0000 7fd8d7167700 -1 osd.2 90 *** Got signal Terminated *** 2024-04-26T21:50:19.291 INFO:journalctl@ceph.osd.2.smithi142.stdout:Apr 26 21:50:18 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-2[37956]: 2024-04-26T21:50:18.953+0000 7fd8d7167700 -1 osd.2 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T21:50:24.739 INFO:journalctl@ceph.osd.2.smithi142.stdout:Apr 26 21:50:24 smithi142 podman[68750]: 2024-04-26 21:50:24.459762757 +0000 UTC m=+5.629548767 container died fe18ac9debe97f6af4f3b6ada8a975b16978d0e4ad7c33d73786da57a471f948 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-04-26T21:50:25.040 INFO:journalctl@ceph.osd.2.smithi142.stdout:Apr 26 21:50:24 smithi142 podman[68750]: 2024-04-26 21:50:24.821573341 +0000 UTC m=+5.991359359 container cleanup fe18ac9debe97f6af4f3b6ada8a975b16978d0e4ad7c33d73786da57a471f948 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux ) 2024-04-26T21:50:25.040 INFO:journalctl@ceph.osd.2.smithi142.stdout:Apr 26 21:50:24 smithi142 bash[68750]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-2 2024-04-26T21:50:25.401 INFO:journalctl@ceph.osd.2.smithi142.stdout:Apr 26 21:50:25 smithi142 podman[68779]: 2024-04-26 21:50:25.089512214 +0000 UTC m=+0.631142189 container remove fe18ac9debe97f6af4f3b6ada8a975b16978d0e4ad7c33d73786da57a471f948 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-2, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-26T21:50:25.654 INFO:journalctl@ceph.osd.2.smithi142.stdout:Apr 26 21:50:25 smithi142 podman[68865]: 2024-04-26 21:50:25.304838699 +0000 UTC m=+0.017826171 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:50:25.654 INFO:journalctl@ceph.osd.2.smithi142.stdout:Apr 26 21:50:25 smithi142 podman[68865]: 2024-04-26 21:50:25.407031972 +0000 UTC m=+0.120019471 container create d4596dea851b2fd25f3dfbf83139b7f73d4b41f4c5d400642ec1b90871e828d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-26T21:50:25.654 INFO:journalctl@ceph.osd.2.smithi142.stdout:Apr 26 21:50:25 smithi142 podman[68865]: 2024-04-26 21:50:25.651217232 +0000 UTC m=+0.364204686 container init d4596dea851b2fd25f3dfbf83139b7f73d4b41f4c5d400642ec1b90871e828d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-26T21:50:25.654 INFO:journalctl@ceph.osd.2.smithi142.stdout:Apr 26 21:50:25 smithi142 podman[68865]: 2024-04-26 21:50:25.654220988 +0000 UTC m=+0.367208442 container start d4596dea851b2fd25f3dfbf83139b7f73d4b41f4c5d400642ec1b90871e828d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-26T21:50:25.946 INFO:journalctl@ceph.osd.2.smithi142.stdout:Apr 26 21:50:25 smithi142 podman[68865]: 2024-04-26 21:50:25.776452382 +0000 UTC m=+0.489439847 container attach d4596dea851b2fd25f3dfbf83139b7f73d4b41f4c5d400642ec1b90871e828d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-26T21:50:26.229 INFO:journalctl@ceph.osd.2.smithi142.stdout:Apr 26 21:50:25 smithi142 podman[68865]: 2024-04-26 21:50:25.947447656 +0000 UTC m=+0.660435112 container died d4596dea851b2fd25f3dfbf83139b7f73d4b41f4c5d400642ec1b90871e828d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-26T21:50:26.545 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.2.service' 2024-04-26T21:50:26.586 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:50:26.586 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-26T21:50:26.586 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-26T21:50:26.586 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.3 2024-04-26T21:50:26.902 INFO:journalctl@ceph.osd.3.smithi046.stdout:Apr 26 21:50:26 smithi046 systemd[1]: Stopping Ceph osd.3 for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:50:27.303 INFO:journalctl@ceph.osd.3.smithi046.stdout:Apr 26 21:50:26 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-3[50586]: 2024-04-26T21:50:26.901+0000 7f13c22b8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:50:27.303 INFO:journalctl@ceph.osd.3.smithi046.stdout:Apr 26 21:50:26 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-3[50586]: 2024-04-26T21:50:26.901+0000 7f13c22b8700 -1 osd.3 90 *** Got signal Terminated *** 2024-04-26T21:50:27.303 INFO:journalctl@ceph.osd.3.smithi046.stdout:Apr 26 21:50:26 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-3[50586]: 2024-04-26T21:50:26.901+0000 7f13c22b8700 -1 osd.3 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T21:50:29.119 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:28 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:28.867+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:29.320 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:28 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:28.921+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:29.394 INFO:journalctl@ceph.osd.3.smithi046.stdout:Apr 26 21:50:29 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-3[50586]: 2024-04-26T21:50:29.215+0000 7f13ba8a4700 -1 osd.3 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.613423+0000 front 2024-04-26T21:50:02.613459+0000 (oldest deadline 2024-04-26T21:50:28.513467+0000) 2024-04-26T21:50:29.803 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:29 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:29.417+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:30.226 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:29 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:29.915+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:30.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:29 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:29.946+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:30.553 INFO:journalctl@ceph.osd.3.smithi046.stdout:Apr 26 21:50:30 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-3[50586]: 2024-04-26T21:50:30.225+0000 7f13ba8a4700 -1 osd.3 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.613423+0000 front 2024-04-26T21:50:02.613459+0000 (oldest deadline 2024-04-26T21:50:28.513467+0000) 2024-04-26T21:50:30.553 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:30 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:30.450+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:30.790 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:30 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:30.452+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:07.488266+0000 front 2024-04-26T21:50:07.488321+0000 (oldest deadline 2024-04-26T21:50:29.788217+0000) 2024-04-26T21:50:31.219 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:30 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:30.906+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:31.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:30 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:30.989+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:31.475 INFO:journalctl@ceph.osd.3.smithi046.stdout:Apr 26 21:50:31 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-3[50586]: 2024-04-26T21:50:31.219+0000 7f13ba8a4700 -1 osd.3 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.613423+0000 front 2024-04-26T21:50:02.613459+0000 (oldest deadline 2024-04-26T21:50:28.513467+0000) 2024-04-26T21:50:31.790 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:31 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:31.437+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:07.488266+0000 front 2024-04-26T21:50:07.488321+0000 (oldest deadline 2024-04-26T21:50:29.788217+0000) 2024-04-26T21:50:31.803 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:31 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:31.475+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:32.289 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:31 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:31.860+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:32.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:31 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:31.993+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:32.394 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:32 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:32.048+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:32.553 INFO:journalctl@ceph.osd.3.smithi046.stdout:Apr 26 21:50:32 smithi046 podman[85017]: 2024-04-26 21:50:32.289739651 +0000 UTC m=+5.504532614 container died 2b88489ed663c034932d27be114f5adbf70f73dfb8865533acf0c202fed5ab9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-3, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-26T21:50:32.553 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:32 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:32.481+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:32.790 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:32 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:32.394+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:07.488266+0000 front 2024-04-26T21:50:07.488321+0000 (oldest deadline 2024-04-26T21:50:29.788217+0000) 2024-04-26T21:50:32.910 INFO:journalctl@ceph.osd.3.smithi046.stdout:Apr 26 21:50:32 smithi046 podman[85017]: 2024-04-26 21:50:32.685349485 +0000 UTC m=+5.900142456 container cleanup 2b88489ed663c034932d27be114f5adbf70f73dfb8865533acf0c202fed5ab9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:50:32.911 INFO:journalctl@ceph.osd.3.smithi046.stdout:Apr 26 21:50:32 smithi046 bash[85017]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-3 2024-04-26T21:50:33.260 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:32 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:32.910+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:33.260 INFO:journalctl@ceph.osd.3.smithi046.stdout:Apr 26 21:50:32 smithi046 podman[85050]: 2024-04-26 21:50:32.961580168 +0000 UTC m=+0.667212248 container remove 2b88489ed663c034932d27be114f5adbf70f73dfb8865533acf0c202fed5ab9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:50:33.290 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:33 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:33.034+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:33.367 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:32 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:32.955+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:33.548 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:33 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:33.436+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:33.548 INFO:journalctl@ceph.osd.3.smithi046.stdout:Apr 26 21:50:33 smithi046 podman[85143]: 2024-04-26 21:50:33.16349128 +0000 UTC m=+0.018497851 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:50:33.548 INFO:journalctl@ceph.osd.3.smithi046.stdout:Apr 26 21:50:33 smithi046 podman[85143]: 2024-04-26 21:50:33.27911851 +0000 UTC m=+0.134125123 container create abfaa376fce6b05475b053849e4bb882e5c61bea5540d66e3c842009a8b54358 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True) 2024-04-26T21:50:33.790 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:33 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:33.413+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:07.488266+0000 front 2024-04-26T21:50:07.488321+0000 (oldest deadline 2024-04-26T21:50:29.788217+0000) 2024-04-26T21:50:33.803 INFO:journalctl@ceph.osd.3.smithi046.stdout:Apr 26 21:50:33 smithi046 podman[85143]: 2024-04-26 21:50:33.548251494 +0000 UTC m=+0.403258066 container init abfaa376fce6b05475b053849e4bb882e5c61bea5540d66e3c842009a8b54358 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-3-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD) 2024-04-26T21:50:33.803 INFO:journalctl@ceph.osd.3.smithi046.stdout:Apr 26 21:50:33 smithi046 podman[85143]: 2024-04-26 21:50:33.552412395 +0000 UTC m=+0.407418970 container start abfaa376fce6b05475b053849e4bb882e5c61bea5540d66e3c842009a8b54358 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-04-26T21:50:33.803 INFO:journalctl@ceph.osd.3.smithi046.stdout:Apr 26 21:50:33 smithi046 podman[85143]: 2024-04-26 21:50:33.623216353 +0000 UTC m=+0.478222927 container attach abfaa376fce6b05475b053849e4bb882e5c61bea5540d66e3c842009a8b54358 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-26T21:50:34.290 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:33 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:33.988+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:34.303 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:33 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:33.934+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:34.303 INFO:journalctl@ceph.osd.3.smithi046.stdout:Apr 26 21:50:33 smithi046 podman[85143]: 2024-04-26 21:50:33.85408455 +0000 UTC m=+0.709091124 container died abfaa376fce6b05475b053849e4bb882e5c61bea5540d66e3c842009a8b54358 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T21:50:34.367 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:33 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:33.908+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:34.643 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:34 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:34.394+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:34.669 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.3.service' 2024-04-26T21:50:34.714 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:50:34.715 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-26T21:50:34.715 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-26T21:50:34.715 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.4 2024-04-26T21:50:34.790 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:34 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:34.382+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:07.488266+0000 front 2024-04-26T21:50:07.488321+0000 (oldest deadline 2024-04-26T21:50:29.788217+0000) 2024-04-26T21:50:35.030 INFO:journalctl@ceph.osd.4.smithi050.stdout:Apr 26 21:50:34 smithi050 systemd[1]: Stopping Ceph osd.4 for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:50:35.031 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:34 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:34.872+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:35.290 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:34 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:34.952+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:35.303 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:34 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:34.974+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:35.367 INFO:journalctl@ceph.osd.4.smithi050.stdout:Apr 26 21:50:35 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-4[44295]: 2024-04-26T21:50:35.030+0000 7f2f94e04700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:50:35.368 INFO:journalctl@ceph.osd.4.smithi050.stdout:Apr 26 21:50:35 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-4[44295]: 2024-04-26T21:50:35.030+0000 7f2f94e04700 -1 osd.4 90 *** Got signal Terminated *** 2024-04-26T21:50:35.368 INFO:journalctl@ceph.osd.4.smithi050.stdout:Apr 26 21:50:35 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-4[44295]: 2024-04-26T21:50:35.030+0000 7f2f94e04700 -1 osd.4 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T21:50:35.790 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:35 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:35.373+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:07.488266+0000 front 2024-04-26T21:50:07.488321+0000 (oldest deadline 2024-04-26T21:50:29.788217+0000) 2024-04-26T21:50:35.803 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:35 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:35.438+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:36.290 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:35 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:35.903+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:36.302 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:36 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:36.019+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:36.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:35 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:35.894+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:36.593 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:36 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:36.330+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:07.488266+0000 front 2024-04-26T21:50:07.488321+0000 (oldest deadline 2024-04-26T21:50:29.788217+0000) 2024-04-26T21:50:36.803 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:36 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:36.420+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:37.290 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:36 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:36.928+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:37.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:36 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:36.908+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:37.470 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:37 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:37.062+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:37.790 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:37 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:37.376+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:07.488266+0000 front 2024-04-26T21:50:07.488321+0000 (oldest deadline 2024-04-26T21:50:29.788217+0000) 2024-04-26T21:50:37.802 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:37 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:37.470+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:37.803 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:37 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:37.470+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.972328+0000 front 2024-04-26T21:50:13.972315+0000 (oldest deadline 2024-04-26T21:50:36.872162+0000) 2024-04-26T21:50:38.290 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:37 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:37.929+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:38.303 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:38 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:38.019+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:38.367 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:37 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:37.930+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:38.790 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:38 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:38.420+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:07.488266+0000 front 2024-04-26T21:50:07.488321+0000 (oldest deadline 2024-04-26T21:50:29.788217+0000) 2024-04-26T21:50:38.803 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:38 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:38.448+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:38.803 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:38 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:38.448+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.972328+0000 front 2024-04-26T21:50:13.972315+0000 (oldest deadline 2024-04-26T21:50:36.872162+0000) 2024-04-26T21:50:39.290 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:38 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:38.910+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:50:39.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:38 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:38.930+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:39.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:38 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:38.930+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:50:39.367 INFO:journalctl@ceph.osd.4.smithi050.stdout:Apr 26 21:50:39 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-4[44295]: 2024-04-26T21:50:39.064+0000 7f2f8d3f0700 -1 osd.4 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:11.870256+0000 front 2024-04-26T21:50:11.870422+0000 (oldest deadline 2024-04-26T21:50:38.870984+0000) 2024-04-26T21:50:39.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:39 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:39.054+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:10.359322+0000 front 2024-04-26T21:50:10.359302+0000 (oldest deadline 2024-04-26T21:50:38.560225+0000) 2024-04-26T21:50:39.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:38 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:38.971+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:39.496 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:39 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:39.059+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:39.790 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:39 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:39.377+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:07.488266+0000 front 2024-04-26T21:50:07.488321+0000 (oldest deadline 2024-04-26T21:50:29.788217+0000) 2024-04-26T21:50:39.790 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:39 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:39.377+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.489618+0000 front 2024-04-26T21:50:14.489603+0000 (oldest deadline 2024-04-26T21:50:38.589517+0000) 2024-04-26T21:50:39.802 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:39 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:39.496+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:39.803 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:39 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:39.496+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.972328+0000 front 2024-04-26T21:50:13.972315+0000 (oldest deadline 2024-04-26T21:50:36.872162+0000) 2024-04-26T21:50:40.290 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:39 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:39.892+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:50:40.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:39 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:39.894+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:40.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:39 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:39.894+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:50:40.303 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:40 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:40.043+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:40.303 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:40 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:40.043+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:50:40.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:40 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:40.030+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:10.359322+0000 front 2024-04-26T21:50:10.359302+0000 (oldest deadline 2024-04-26T21:50:38.560225+0000) 2024-04-26T21:50:40.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:39 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:39.973+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:40.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:39 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:39.973+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:50:40.790 INFO:journalctl@ceph.osd.4.smithi050.stdout:Apr 26 21:50:40 smithi050 podman[67813]: 2024-04-26 21:50:40.492975377 +0000 UTC m=+5.580129299 container died d3b2bedf1fb1e7df4fbc60ddef8c49b238b6228175f663fc91bb06bcf3968517 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-26T21:50:40.790 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:40 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:40.357+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:07.488266+0000 front 2024-04-26T21:50:07.488321+0000 (oldest deadline 2024-04-26T21:50:29.788217+0000) 2024-04-26T21:50:40.790 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:40 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:40.357+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.489618+0000 front 2024-04-26T21:50:14.489603+0000 (oldest deadline 2024-04-26T21:50:38.589517+0000) 2024-04-26T21:50:40.803 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:40 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:40.499+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:40.803 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:40 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:40.499+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.972328+0000 front 2024-04-26T21:50:13.972315+0000 (oldest deadline 2024-04-26T21:50:36.872162+0000) 2024-04-26T21:50:41.043 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:41 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:41.020+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:41.043 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:41 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:41.020+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:50:41.043 INFO:journalctl@ceph.osd.4.smithi050.stdout:Apr 26 21:50:40 smithi050 podman[67813]: 2024-04-26 21:50:40.88081719 +0000 UTC m=+5.967971143 container cleanup d3b2bedf1fb1e7df4fbc60ddef8c49b238b6228175f663fc91bb06bcf3968517 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-4, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-26T21:50:41.044 INFO:journalctl@ceph.osd.4.smithi050.stdout:Apr 26 21:50:40 smithi050 bash[67813]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-4 2024-04-26T21:50:41.290 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:40 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:40.904+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:50:41.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:40 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:40.869+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:41.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:40 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:40.869+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:50:41.302 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:41 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:41.040+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:41.303 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:41 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:41.040+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:50:41.367 INFO:journalctl@ceph.osd.4.smithi050.stdout:Apr 26 21:50:41 smithi050 podman[67830]: 2024-04-26 21:50:41.10721289 +0000 UTC m=+0.612693616 container remove d3b2bedf1fb1e7df4fbc60ddef8c49b238b6228175f663fc91bb06bcf3968517 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:50:41.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:41 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:41.042+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:10.359322+0000 front 2024-04-26T21:50:10.359302+0000 (oldest deadline 2024-04-26T21:50:38.560225+0000) 2024-04-26T21:50:41.790 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:41 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:41.314+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:07.488266+0000 front 2024-04-26T21:50:07.488321+0000 (oldest deadline 2024-04-26T21:50:29.788217+0000) 2024-04-26T21:50:42.303 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:41 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:41.314+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.489618+0000 front 2024-04-26T21:50:14.489603+0000 (oldest deadline 2024-04-26T21:50:38.589517+0000) 2024-04-26T21:50:42.305 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:41 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:41.536+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:42.305 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:41 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:41.536+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.972328+0000 front 2024-04-26T21:50:13.972315+0000 (oldest deadline 2024-04-26T21:50:36.872162+0000) 2024-04-26T21:50:42.305 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:41 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:41.858+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:50:42.305 INFO:journalctl@ceph.osd.4.smithi050.stdout:Apr 26 21:50:41 smithi050 podman[67920]: 2024-04-26 21:50:41.420013752 +0000 UTC m=+0.018707450 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:50:42.305 INFO:journalctl@ceph.osd.4.smithi050.stdout:Apr 26 21:50:41 smithi050 podman[67920]: 2024-04-26 21:50:41.533499328 +0000 UTC m=+0.132193056 container create 5a6ba824b12c2b7403ed2b01d37ad9eca1490d710cef0ddce1eb9d8988f1b130 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-4-deactivate, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-26T21:50:42.305 INFO:journalctl@ceph.osd.4.smithi050.stdout:Apr 26 21:50:41 smithi050 podman[67920]: 2024-04-26 21:50:41.794592126 +0000 UTC m=+0.393285813 container init 5a6ba824b12c2b7403ed2b01d37ad9eca1490d710cef0ddce1eb9d8988f1b130 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-26T21:50:42.306 INFO:journalctl@ceph.osd.4.smithi050.stdout:Apr 26 21:50:41 smithi050 podman[67920]: 2024-04-26 21:50:41.798322498 +0000 UTC m=+0.397016192 container start 5a6ba824b12c2b7403ed2b01d37ad9eca1490d710cef0ddce1eb9d8988f1b130 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T21:50:42.306 INFO:journalctl@ceph.osd.4.smithi050.stdout:Apr 26 21:50:41 smithi050 podman[67920]: 2024-04-26 21:50:41.894508405 +0000 UTC m=+0.493202099 container attach 5a6ba824b12c2b7403ed2b01d37ad9eca1490d710cef0ddce1eb9d8988f1b130 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T21:50:42.306 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:41 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:41.888+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:42.306 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:41 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:41.888+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:50:42.367 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:42 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:42.085+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:10.359322+0000 front 2024-04-26T21:50:10.359302+0000 (oldest deadline 2024-04-26T21:50:38.560225+0000) 2024-04-26T21:50:42.368 INFO:journalctl@ceph.osd.4.smithi050.stdout:Apr 26 21:50:42 smithi050 podman[67920]: 2024-04-26 21:50:42.105297621 +0000 UTC m=+0.703991308 container died 5a6ba824b12c2b7403ed2b01d37ad9eca1490d710cef0ddce1eb9d8988f1b130 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-26T21:50:42.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:42 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:42.065+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:42.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:42 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:42.065+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:50:42.525 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:42 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:42.073+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:42.525 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:42 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:42.073+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:50:42.763 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.4.service' 2024-04-26T21:50:42.790 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:42 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:42.315+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:07.488266+0000 front 2024-04-26T21:50:07.488321+0000 (oldest deadline 2024-04-26T21:50:29.788217+0000) 2024-04-26T21:50:42.790 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:42 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:42.315+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.489618+0000 front 2024-04-26T21:50:14.489603+0000 (oldest deadline 2024-04-26T21:50:38.589517+0000) 2024-04-26T21:50:42.802 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:42 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:42.525+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:42.803 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:42 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:42.525+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.972328+0000 front 2024-04-26T21:50:13.972315+0000 (oldest deadline 2024-04-26T21:50:36.872162+0000) 2024-04-26T21:50:42.809 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:50:42.809 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-26T21:50:42.809 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-26T21:50:42.809 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.5 2024-04-26T21:50:43.090 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:42 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:42.822+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:50:43.091 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:42 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:42.935+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:43.091 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:42 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:42.935+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:50:43.091 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:42 smithi142 systemd[1]: Stopping Ceph osd.5 for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:50:43.091 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:43 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:43.090+0000 7f013e017700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:50:43.091 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:43 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:43.090+0000 7f013e017700 -1 osd.5 90 *** Got signal Terminated *** 2024-04-26T21:50:43.092 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:43 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:43.090+0000 7f013e017700 -1 osd.5 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T21:50:43.367 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:43 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:43.128+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:10.359322+0000 front 2024-04-26T21:50:10.359302+0000 (oldest deadline 2024-04-26T21:50:38.560225+0000) 2024-04-26T21:50:43.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:43 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:43.022+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:43.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:43 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:43.022+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:50:43.528 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:43 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:43.110+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:43.529 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:43 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:43.110+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:50:43.540 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:43 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:43.321+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:07.488266+0000 front 2024-04-26T21:50:07.488321+0000 (oldest deadline 2024-04-26T21:50:29.788217+0000) 2024-04-26T21:50:43.540 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:43 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:43.321+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.489618+0000 front 2024-04-26T21:50:14.489603+0000 (oldest deadline 2024-04-26T21:50:38.589517+0000) 2024-04-26T21:50:43.803 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:43 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:43.529+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:43.803 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:43 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:43.529+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.972328+0000 front 2024-04-26T21:50:13.972315+0000 (oldest deadline 2024-04-26T21:50:36.872162+0000) 2024-04-26T21:50:44.290 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:43 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:43.860+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:50:44.290 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:43 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:43.888+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:44.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:43 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:43.888+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:50:44.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:44 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:44.155+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:10.359322+0000 front 2024-04-26T21:50:10.359302+0000 (oldest deadline 2024-04-26T21:50:38.560225+0000) 2024-04-26T21:50:44.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:44 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:44.061+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:44.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:44 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:44.061+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:50:44.479 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:44 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:44.077+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:44.479 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:44 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:44.077+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:50:44.758 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:44 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:44.479+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:44.759 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:44 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:44.479+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.972328+0000 front 2024-04-26T21:50:13.972315+0000 (oldest deadline 2024-04-26T21:50:36.872162+0000) 2024-04-26T21:50:44.790 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:44 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:44.358+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:07.488266+0000 front 2024-04-26T21:50:07.488321+0000 (oldest deadline 2024-04-26T21:50:29.788217+0000) 2024-04-26T21:50:44.790 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:44 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:44.358+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.489618+0000 front 2024-04-26T21:50:14.489603+0000 (oldest deadline 2024-04-26T21:50:38.589517+0000) 2024-04-26T21:50:45.290 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:44 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:44.858+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:50:45.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:44 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:44.937+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:45.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:44 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:44.937+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:50:45.367 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:45 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:45.133+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:10.359322+0000 front 2024-04-26T21:50:10.359302+0000 (oldest deadline 2024-04-26T21:50:38.560225+0000) 2024-04-26T21:50:45.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:45 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:45.039+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:45.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:45 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:45.039+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:50:45.430 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:45 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:45.109+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:45.430 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:45 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:45.109+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:50:45.430 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:45 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:45.109+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:50:45.790 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:45 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:45.396+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:07.488266+0000 front 2024-04-26T21:50:07.488321+0000 (oldest deadline 2024-04-26T21:50:29.788217+0000) 2024-04-26T21:50:45.790 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:45 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:45.396+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.489618+0000 front 2024-04-26T21:50:14.489603+0000 (oldest deadline 2024-04-26T21:50:38.589517+0000) 2024-04-26T21:50:45.803 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:45 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:45.430+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:45.803 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:45 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:45.430+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.972328+0000 front 2024-04-26T21:50:13.972315+0000 (oldest deadline 2024-04-26T21:50:36.872162+0000) 2024-04-26T21:50:45.803 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:45 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:45.430+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.173247+0000 front 2024-04-26T21:50:22.173475+0000 (oldest deadline 2024-04-26T21:50:45.073228+0000) 2024-04-26T21:50:46.290 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:45 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:45.842+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:50:46.290 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:45 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:45.842+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:50:46.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:45 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:45.907+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:46.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:45 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:45.907+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:50:46.367 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:46 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:46.176+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:10.359322+0000 front 2024-04-26T21:50:10.359302+0000 (oldest deadline 2024-04-26T21:50:38.560225+0000) 2024-04-26T21:50:46.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:46 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:46.077+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:46.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:46 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:46.077+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:50:46.553 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:46 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:46.402+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:46.553 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:46 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:46.402+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.972328+0000 front 2024-04-26T21:50:13.972315+0000 (oldest deadline 2024-04-26T21:50:36.872162+0000) 2024-04-26T21:50:46.553 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:46 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:46.402+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.173247+0000 front 2024-04-26T21:50:22.173475+0000 (oldest deadline 2024-04-26T21:50:45.073228+0000) 2024-04-26T21:50:46.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:46 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:46.158+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:46.554 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:46 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:46.158+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:50:46.554 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:46 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:46.158+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:50:46.790 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:46 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:46.364+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:07.488266+0000 front 2024-04-26T21:50:07.488321+0000 (oldest deadline 2024-04-26T21:50:29.788217+0000) 2024-04-26T21:50:46.790 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:46 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:46.364+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.489618+0000 front 2024-04-26T21:50:14.489603+0000 (oldest deadline 2024-04-26T21:50:38.589517+0000) 2024-04-26T21:50:46.790 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:46 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:46.364+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.890538+0000 front 2024-04-26T21:50:23.890562+0000 (oldest deadline 2024-04-26T21:50:45.491337+0000) 2024-04-26T21:50:47.290 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:46 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:46.818+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:50:47.291 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:46 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:46.818+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:50:47.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:46 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:46.880+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:47.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:46 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:46.880+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:50:47.367 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:47 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:47.166+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:10.359322+0000 front 2024-04-26T21:50:10.359302+0000 (oldest deadline 2024-04-26T21:50:38.560225+0000) 2024-04-26T21:50:47.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:47 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:47.048+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:47.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:47 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:47.048+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:50:47.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:47 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:47.180+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:47.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:47 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:47.180+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:50:47.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:47 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:47.180+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:50:47.554 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:47 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:47.417+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:47.554 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:47 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:47.417+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.972328+0000 front 2024-04-26T21:50:13.972315+0000 (oldest deadline 2024-04-26T21:50:36.872162+0000) 2024-04-26T21:50:47.554 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:47 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:47.417+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.173247+0000 front 2024-04-26T21:50:22.173475+0000 (oldest deadline 2024-04-26T21:50:45.073228+0000) 2024-04-26T21:50:47.791 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:47 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:47.378+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:07.488266+0000 front 2024-04-26T21:50:07.488321+0000 (oldest deadline 2024-04-26T21:50:29.788217+0000) 2024-04-26T21:50:47.791 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:47 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:47.378+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.489618+0000 front 2024-04-26T21:50:14.489603+0000 (oldest deadline 2024-04-26T21:50:38.589517+0000) 2024-04-26T21:50:47.791 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:47 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5[44875]: 2024-04-26T21:50:47.378+0000 7f0136603700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.890538+0000 front 2024-04-26T21:50:23.890562+0000 (oldest deadline 2024-04-26T21:50:45.491337+0000) 2024-04-26T21:50:48.290 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:47 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:47.836+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:50:48.290 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:47 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:47.836+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:50:48.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:47 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:47.888+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:48.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:47 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:47.888+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:50:48.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:48 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:48.128+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:10.359322+0000 front 2024-04-26T21:50:10.359302+0000 (oldest deadline 2024-04-26T21:50:38.560225+0000) 2024-04-26T21:50:48.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:48 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:48.002+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:48.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:48 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:48.002+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:50:48.553 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:48 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:48.379+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:48.553 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:48 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:48.379+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.972328+0000 front 2024-04-26T21:50:13.972315+0000 (oldest deadline 2024-04-26T21:50:36.872162+0000) 2024-04-26T21:50:48.553 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:48 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:48.379+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.173247+0000 front 2024-04-26T21:50:22.173475+0000 (oldest deadline 2024-04-26T21:50:45.073228+0000) 2024-04-26T21:50:48.554 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:48 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:48.229+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:48.554 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:48 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:48.229+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:50:48.554 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:48 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:48.229+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:50:48.766 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:48 smithi142 podman[68971]: 2024-04-26 21:50:48.460605555 +0000 UTC m=+5.460133763 container died fb8f44a8a152b8e935723bdc5d6edec9215469a54f683f07f8116e7f6ae5f739 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T21:50:49.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:48 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:48.830+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:50:49.041 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:48 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:48.830+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:50:49.041 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:48 smithi142 podman[68971]: 2024-04-26 21:50:48.857733055 +0000 UTC m=+5.857261259 container cleanup fb8f44a8a152b8e935723bdc5d6edec9215469a54f683f07f8116e7f6ae5f739 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-26T21:50:49.041 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:48 smithi142 bash[68971]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5 2024-04-26T21:50:49.042 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:48 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:48.863+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:49.042 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:48 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:48.863+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:50:49.367 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:49 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:49.086+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:10.359322+0000 front 2024-04-26T21:50:10.359302+0000 (oldest deadline 2024-04-26T21:50:38.560225+0000) 2024-04-26T21:50:49.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:48 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:48.976+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:49.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:48 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:48.976+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:50:49.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:48 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:48.976+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:50:49.424 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:49 smithi142 podman[68990]: 2024-04-26 21:50:49.100653582 +0000 UTC m=+0.634203323 container remove fb8f44a8a152b8e935723bdc5d6edec9215469a54f683f07f8116e7f6ae5f739 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-26T21:50:49.553 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:49 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:49.359+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:49.553 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:49 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:49.359+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.972328+0000 front 2024-04-26T21:50:13.972315+0000 (oldest deadline 2024-04-26T21:50:36.872162+0000) 2024-04-26T21:50:49.553 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:49 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:49.359+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.173247+0000 front 2024-04-26T21:50:22.173475+0000 (oldest deadline 2024-04-26T21:50:45.073228+0000) 2024-04-26T21:50:49.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:49 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:49.197+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:49.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:49 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:49.197+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:50:49.554 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:49 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:49.197+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:50:49.712 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:49 smithi142 podman[69081]: 2024-04-26 21:50:49.328105374 +0000 UTC m=+0.018058648 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:50:49.712 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:49 smithi142 podman[69081]: 2024-04-26 21:50:49.459879909 +0000 UTC m=+0.149833145 container create b00ef1e106c464ced4196282b333154ecc4452c03261697d3a57f920d5bc0b58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0) 2024-04-26T21:50:49.712 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:49 smithi142 podman[69081]: 2024-04-26 21:50:49.712346693 +0000 UTC m=+0.402299953 container init b00ef1e106c464ced4196282b333154ecc4452c03261697d3a57f920d5bc0b58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T21:50:50.002 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:49 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:49.848+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:50:50.002 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:49 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:49.848+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:50:50.003 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:49 smithi142 podman[69081]: 2024-04-26 21:50:49.715766548 +0000 UTC m=+0.405719786 container start b00ef1e106c464ced4196282b333154ecc4452c03261697d3a57f920d5bc0b58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=) 2024-04-26T21:50:50.003 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:49 smithi142 podman[69081]: 2024-04-26 21:50:49.795592222 +0000 UTC m=+0.485545475 container attach b00ef1e106c464ced4196282b333154ecc4452c03261697d3a57f920d5bc0b58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2) 2024-04-26T21:50:50.003 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:49 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:49.838+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:50.003 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:49 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:49.838+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:50:50.004 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:49 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:49.838+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:50:50.275 INFO:journalctl@ceph.osd.5.smithi142.stdout:Apr 26 21:50:50 smithi142 podman[69081]: 2024-04-26 21:50:50.003058585 +0000 UTC m=+0.693011841 container died b00ef1e106c464ced4196282b333154ecc4452c03261697d3a57f920d5bc0b58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-5-deactivate, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-26T21:50:50.367 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:49 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:49.964+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:50.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:49 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:49.964+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:50:50.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:49 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:49.964+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:50:50.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:50 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:50.062+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:10.359322+0000 front 2024-04-26T21:50:10.359302+0000 (oldest deadline 2024-04-26T21:50:38.560225+0000) 2024-04-26T21:50:50.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:50 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:50.062+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:21.361781+0000 front 2024-04-26T21:50:21.361918+0000 (oldest deadline 2024-04-26T21:50:49.562305+0000) 2024-04-26T21:50:50.553 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:50 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:50.390+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:50.553 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:50 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:50.390+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.972328+0000 front 2024-04-26T21:50:13.972315+0000 (oldest deadline 2024-04-26T21:50:36.872162+0000) 2024-04-26T21:50:50.553 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:50 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:50.390+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.173247+0000 front 2024-04-26T21:50:22.173475+0000 (oldest deadline 2024-04-26T21:50:45.073228+0000) 2024-04-26T21:50:50.554 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:50 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:50.179+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:50.554 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:50 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:50.179+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:50:50.554 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:50 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:50.179+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:50:50.556 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.5.service' 2024-04-26T21:50:50.593 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:50:50.593 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-26T21:50:50.594 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-26T21:50:50.594 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.6 2024-04-26T21:50:51.004 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:50 smithi046 systemd[1]: Stopping Ceph osd.6 for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:50:51.213 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:50 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:50.832+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:50:51.214 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:50 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:50.832+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:50:51.214 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:50 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:50.872+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:51.214 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:50 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:50.872+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:50:51.214 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:50 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:50.872+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:50:51.303 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:51 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:51.004+0000 7f0c1d37e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:50:51.303 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:51 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:51.004+0000 7f0c1d37e700 -1 osd.6 90 *** Got signal Terminated *** 2024-04-26T21:50:51.303 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:51 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:51.004+0000 7f0c1d37e700 -1 osd.6 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T21:50:51.304 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:51 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:51.154+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:51.304 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:51 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:51.154+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:50:51.304 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:51 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:51.154+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:50:51.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:51 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:51.079+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:10.359322+0000 front 2024-04-26T21:50:10.359302+0000 (oldest deadline 2024-04-26T21:50:38.560225+0000) 2024-04-26T21:50:51.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:51 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:51.079+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:21.361781+0000 front 2024-04-26T21:50:21.361918+0000 (oldest deadline 2024-04-26T21:50:49.562305+0000) 2024-04-26T21:50:51.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:50 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:50.993+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:51.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:50 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:50.993+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:50:51.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:50 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:50.993+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:50:51.803 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:51 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:51.388+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:51.803 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:51 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:51.388+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.972328+0000 front 2024-04-26T21:50:13.972315+0000 (oldest deadline 2024-04-26T21:50:36.872162+0000) 2024-04-26T21:50:51.803 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:51 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:51.388+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.173247+0000 front 2024-04-26T21:50:22.173475+0000 (oldest deadline 2024-04-26T21:50:45.073228+0000) 2024-04-26T21:50:52.290 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:51 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:51.858+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:50:52.291 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:51 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:51.858+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:50:52.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:51 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:51.918+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:52.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:51 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:51.918+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:50:52.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:51 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:51.918+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:50:52.367 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:52 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:52.129+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:10.359322+0000 front 2024-04-26T21:50:10.359302+0000 (oldest deadline 2024-04-26T21:50:38.560225+0000) 2024-04-26T21:50:52.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:52 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:52.129+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:21.361781+0000 front 2024-04-26T21:50:21.361918+0000 (oldest deadline 2024-04-26T21:50:49.562305+0000) 2024-04-26T21:50:52.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:51 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:51.959+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:52.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:51 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:51.959+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:50:52.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:51 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:51.959+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:50:52.553 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:52 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:52.411+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:52.553 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:52 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:52.411+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.972328+0000 front 2024-04-26T21:50:13.972315+0000 (oldest deadline 2024-04-26T21:50:36.872162+0000) 2024-04-26T21:50:52.553 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:52 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:52.411+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.173247+0000 front 2024-04-26T21:50:22.173475+0000 (oldest deadline 2024-04-26T21:50:45.073228+0000) 2024-04-26T21:50:52.554 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:52 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:52.164+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:52.554 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:52 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:52.164+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:50:52.554 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:52 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:52.164+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:50:53.290 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:52 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:52.858+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:50:53.290 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:52 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:52.858+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:50:53.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:52 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:52.958+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:53.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:52 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:52.958+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:50:53.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:52 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:52.958+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:50:53.370 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:53 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:53.145+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:10.359322+0000 front 2024-04-26T21:50:10.359302+0000 (oldest deadline 2024-04-26T21:50:38.560225+0000) 2024-04-26T21:50:53.370 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:53 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:53.145+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:21.361781+0000 front 2024-04-26T21:50:21.361918+0000 (oldest deadline 2024-04-26T21:50:49.562305+0000) 2024-04-26T21:50:53.370 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:52 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:52.961+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:53.371 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:52 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:52.961+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:50:53.371 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:52 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:52.961+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:50:53.553 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:53 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:53.374+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:53.553 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:53 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:53.374+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.972328+0000 front 2024-04-26T21:50:13.972315+0000 (oldest deadline 2024-04-26T21:50:36.872162+0000) 2024-04-26T21:50:53.553 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:53 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:53.374+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.173247+0000 front 2024-04-26T21:50:22.173475+0000 (oldest deadline 2024-04-26T21:50:45.073228+0000) 2024-04-26T21:50:53.554 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:53 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:53.140+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:53.554 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:53 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:53.140+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:50:53.554 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:53 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:53.140+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:50:53.554 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:53 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:53.140+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:50:54.290 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:53 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:53.836+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:50:54.290 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:53 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:53.836+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:50:54.291 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:53 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:53.836+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:30.704083+0000 front 2024-04-26T21:50:30.703944+0000 (oldest deadline 2024-04-26T21:50:53.004036+0000) 2024-04-26T21:50:54.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:54 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:54.008+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:54.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:54 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:54.008+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:50:54.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:54 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:54.008+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:50:54.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:54 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:54.141+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:10.359322+0000 front 2024-04-26T21:50:10.359302+0000 (oldest deadline 2024-04-26T21:50:38.560225+0000) 2024-04-26T21:50:54.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:54 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:54.141+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:21.361781+0000 front 2024-04-26T21:50:21.361918+0000 (oldest deadline 2024-04-26T21:50:49.562305+0000) 2024-04-26T21:50:54.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:53 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:53.964+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:54.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:53 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:53.964+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:50:54.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:53 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:53.964+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:50:54.392 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:54 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:54.113+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:54.392 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:54 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:54.113+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:50:54.392 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:54 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:54.113+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:50:54.393 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:54 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:54.113+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:50:54.803 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:54 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:54.391+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:54.803 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:54 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:54.391+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.972328+0000 front 2024-04-26T21:50:13.972315+0000 (oldest deadline 2024-04-26T21:50:36.872162+0000) 2024-04-26T21:50:54.803 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:54 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:54.391+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.173247+0000 front 2024-04-26T21:50:22.173475+0000 (oldest deadline 2024-04-26T21:50:45.073228+0000) 2024-04-26T21:50:55.290 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:54 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:54.862+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:50:55.290 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:54 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:54.862+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:50:55.290 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:54 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:54.862+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:30.704083+0000 front 2024-04-26T21:50:30.703944+0000 (oldest deadline 2024-04-26T21:50:53.004036+0000) 2024-04-26T21:50:55.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:55 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:55.030+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:55.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:55 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:55.030+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:50:55.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:55 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:55.030+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:50:55.346 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:55 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:55.065+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:55.346 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:55 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:55.065+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:50:55.346 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:55 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:55.065+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:50:55.346 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:55 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:55.065+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:50:55.367 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:55 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:55.121+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:10.359322+0000 front 2024-04-26T21:50:10.359302+0000 (oldest deadline 2024-04-26T21:50:38.560225+0000) 2024-04-26T21:50:55.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:55 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:55.121+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:21.361781+0000 front 2024-04-26T21:50:21.361918+0000 (oldest deadline 2024-04-26T21:50:49.562305+0000) 2024-04-26T21:50:55.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:55 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:55.121+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.562999+0000 front 2024-04-26T21:50:29.562945+0000 (oldest deadline 2024-04-26T21:50:54.262883+0000) 2024-04-26T21:50:55.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:54 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:54.987+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:55.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:54 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:54.987+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:50:55.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:54 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:54.987+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:50:55.803 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:55 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:55.345+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:02.770982+0000 front 2024-04-26T21:50:02.771160+0000 (oldest deadline 2024-04-26T21:50:28.671103+0000) 2024-04-26T21:50:55.803 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:55 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:55.345+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.972328+0000 front 2024-04-26T21:50:13.972315+0000 (oldest deadline 2024-04-26T21:50:36.872162+0000) 2024-04-26T21:50:55.803 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:55 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6[57167]: 2024-04-26T21:50:55.345+0000 7f0c19972700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.173247+0000 front 2024-04-26T21:50:22.173475+0000 (oldest deadline 2024-04-26T21:50:45.073228+0000) 2024-04-26T21:50:56.291 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:55 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:55.828+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:50:56.291 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:55 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:55.828+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:50:56.291 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:55 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:55.828+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:30.704083+0000 front 2024-04-26T21:50:30.703944+0000 (oldest deadline 2024-04-26T21:50:53.004036+0000) 2024-04-26T21:50:56.292 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:56 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:56.004+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:56.292 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:56 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:56.004+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:50:56.292 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:56 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:56.004+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:50:56.292 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:56 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:56.004+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:50:56.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:56 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:56.134+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:10.359322+0000 front 2024-04-26T21:50:10.359302+0000 (oldest deadline 2024-04-26T21:50:38.560225+0000) 2024-04-26T21:50:56.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:56 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:56.134+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:21.361781+0000 front 2024-04-26T21:50:21.361918+0000 (oldest deadline 2024-04-26T21:50:49.562305+0000) 2024-04-26T21:50:56.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:56 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:56.134+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.562999+0000 front 2024-04-26T21:50:29.562945+0000 (oldest deadline 2024-04-26T21:50:54.262883+0000) 2024-04-26T21:50:56.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:55 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:55.988+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:56.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:55 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:55.988+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:50:56.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:55 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:55.988+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:50:56.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:55 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:55.988+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:50:56.553 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:56 smithi046 podman[85247]: 2024-04-26 21:50:56.334181939 +0000 UTC m=+5.541093887 container died 8d397069bb8203d39f6c7eb71784a669fa5e33555d1972d242d86eafba918e78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.build-date=20240424) 2024-04-26T21:50:56.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:56 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:56.092+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:56.554 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:56 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:56.092+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:50:56.554 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:56 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:56.092+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:50:56.554 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:56 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:56.092+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:50:57.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:56 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:56.779+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:50:57.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:56 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:56.779+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:50:57.041 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:56 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:56.779+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:30.704083+0000 front 2024-04-26T21:50:30.703944+0000 (oldest deadline 2024-04-26T21:50:53.004036+0000) 2024-04-26T21:50:57.041 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:57 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:57.026+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:57.041 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:57 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:57.026+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:50:57.041 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:57 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:57.026+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:50:57.041 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:57 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:57.026+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:50:57.052 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:56 smithi046 podman[85247]: 2024-04-26 21:50:56.695571574 +0000 UTC m=+5.902483523 container cleanup 8d397069bb8203d39f6c7eb71784a669fa5e33555d1972d242d86eafba918e78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-26T21:50:57.053 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:56 smithi046 bash[85247]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6 2024-04-26T21:50:57.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:57 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:57.101+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:10.359322+0000 front 2024-04-26T21:50:10.359302+0000 (oldest deadline 2024-04-26T21:50:38.560225+0000) 2024-04-26T21:50:57.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:57 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:57.101+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:21.361781+0000 front 2024-04-26T21:50:21.361918+0000 (oldest deadline 2024-04-26T21:50:49.562305+0000) 2024-04-26T21:50:57.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:57 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:57.101+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.562999+0000 front 2024-04-26T21:50:29.562945+0000 (oldest deadline 2024-04-26T21:50:54.262883+0000) 2024-04-26T21:50:57.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:56 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:56.954+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:57.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:56 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:56.954+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:50:57.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:56 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:56.954+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:50:57.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:56 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:56.954+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:50:57.446 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:57 smithi046 podman[85265]: 2024-04-26 21:50:57.121868716 +0000 UTC m=+0.778505135 container remove 8d397069bb8203d39f6c7eb71784a669fa5e33555d1972d242d86eafba918e78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-26T21:50:57.446 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:57 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:57.138+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:57.446 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:57 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:57.138+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:50:57.446 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:57 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:57.138+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:50:57.447 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:57 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:57.138+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:50:57.717 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:57 smithi046 podman[85357]: 2024-04-26 21:50:57.349528942 +0000 UTC m=+0.018292654 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:50:57.718 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:57 smithi046 podman[85357]: 2024-04-26 21:50:57.472544655 +0000 UTC m=+0.141308400 container create e422ec8f854e5bd319ca97e39133af8f57abfcf81bf6638cde8d6263c246ed7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424) 2024-04-26T21:50:58.023 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:57 smithi046 podman[85357]: 2024-04-26 21:50:57.717934989 +0000 UTC m=+0.386698694 container init e422ec8f854e5bd319ca97e39133af8f57abfcf81bf6638cde8d6263c246ed7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-26T21:50:58.024 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:57 smithi046 podman[85357]: 2024-04-26 21:50:57.721491184 +0000 UTC m=+0.390254889 container start e422ec8f854e5bd319ca97e39133af8f57abfcf81bf6638cde8d6263c246ed7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-26T21:50:58.024 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:57 smithi046 podman[85357]: 2024-04-26 21:50:57.816787263 +0000 UTC m=+0.485550984 container attach e422ec8f854e5bd319ca97e39133af8f57abfcf81bf6638cde8d6263c246ed7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-26T21:50:58.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:57 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:57.748+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:50:58.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:57 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:57.748+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:50:58.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:57 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:57.748+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:30.704083+0000 front 2024-04-26T21:50:30.703944+0000 (oldest deadline 2024-04-26T21:50:53.004036+0000) 2024-04-26T21:50:58.303 INFO:journalctl@ceph.osd.6.smithi046.stdout:Apr 26 21:50:58 smithi046 podman[85357]: 2024-04-26 21:50:58.023977827 +0000 UTC m=+0.692741534 container died e422ec8f854e5bd319ca97e39133af8f57abfcf81bf6638cde8d6263c246ed7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:50:58.303 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:58 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:58.181+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:58.304 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:58 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:58.181+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:50:58.304 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:58 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:58.181+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:50:58.304 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:58 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:58.181+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:50:58.367 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:58 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:58.059+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:10.359322+0000 front 2024-04-26T21:50:10.359302+0000 (oldest deadline 2024-04-26T21:50:38.560225+0000) 2024-04-26T21:50:58.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:58 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:58.059+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:21.361781+0000 front 2024-04-26T21:50:21.361918+0000 (oldest deadline 2024-04-26T21:50:49.562305+0000) 2024-04-26T21:50:58.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:58 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:58.059+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.562999+0000 front 2024-04-26T21:50:29.562945+0000 (oldest deadline 2024-04-26T21:50:54.262883+0000) 2024-04-26T21:50:58.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:57 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:57.955+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:58.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:57 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:57.955+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:50:58.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:57 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:57.955+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:50:58.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:57 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:57.955+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:50:58.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:58 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:58.065+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:58.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:58 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:58.065+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:50:58.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:58 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:58.065+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:50:58.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:58 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:58.065+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:50:58.642 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.6.service' 2024-04-26T21:50:58.682 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:50:58.682 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-26T21:50:58.683 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-26T21:50:58.683 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.7 2024-04-26T21:50:59.001 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:58 smithi050 systemd[1]: Stopping Ceph osd.7 for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:50:59.001 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:59 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:59.000+0000 7fb1e952a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:50:59.001 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:59 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:59.000+0000 7fb1e952a700 -1 osd.7 90 *** Got signal Terminated *** 2024-04-26T21:50:59.002 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:58 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:58.948+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:50:59.002 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:58 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:58.948+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:50:59.002 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:58 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:58.948+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:50:59.002 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:58 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:58.948+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:50:59.018 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:58 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:58.752+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:50:59.018 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:58 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:58.752+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:50:59.018 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:58 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:58.752+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:30.704083+0000 front 2024-04-26T21:50:30.703944+0000 (oldest deadline 2024-04-26T21:50:53.004036+0000) 2024-04-26T21:50:59.290 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:59 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:59.018+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:50:59.290 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:59 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:59.018+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:50:59.290 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:59 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:59.018+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:50:59.291 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:50:59 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:50:59.018+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:50:59.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:59 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:59.000+0000 7fb1e952a700 -1 osd.7 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T21:50:59.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:59 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:59.096+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:10.359322+0000 front 2024-04-26T21:50:10.359302+0000 (oldest deadline 2024-04-26T21:50:38.560225+0000) 2024-04-26T21:50:59.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:59 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:59.096+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:21.361781+0000 front 2024-04-26T21:50:21.361918+0000 (oldest deadline 2024-04-26T21:50:49.562305+0000) 2024-04-26T21:50:59.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:50:59 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:50:59.096+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.562999+0000 front 2024-04-26T21:50:29.562945+0000 (oldest deadline 2024-04-26T21:50:54.262883+0000) 2024-04-26T21:50:59.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:59 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:59.231+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:50:59.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:59 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:59.231+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:50:59.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:59 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:59.231+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:50:59.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:50:59 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:50:59.231+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:51:00.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:59 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:59.722+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:51:00.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:59 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:59.722+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:51:00.041 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:50:59 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:50:59.722+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:30.704083+0000 front 2024-04-26T21:50:30.703944+0000 (oldest deadline 2024-04-26T21:50:53.004036+0000) 2024-04-26T21:51:00.343 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:00 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:00.232+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:51:00.343 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:00 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:00.232+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:51:00.344 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:00 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:00.232+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:51:00.344 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:00 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:00.232+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:51:00.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:51:00 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:51:00.083+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:10.359322+0000 front 2024-04-26T21:50:10.359302+0000 (oldest deadline 2024-04-26T21:50:38.560225+0000) 2024-04-26T21:51:00.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:51:00 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:51:00.083+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:21.361781+0000 front 2024-04-26T21:50:21.361918+0000 (oldest deadline 2024-04-26T21:50:49.562305+0000) 2024-04-26T21:51:00.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:51:00 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:51:00.083+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.562999+0000 front 2024-04-26T21:50:29.562945+0000 (oldest deadline 2024-04-26T21:50:54.262883+0000) 2024-04-26T21:51:00.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:59 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:59.973+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:00.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:59 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:59.973+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:00.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:59 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:59.973+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:00.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:50:59 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:50:59.973+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:00.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:00 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:00.055+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:00.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:00 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:00.055+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:00.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:00 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:00.055+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:00.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:00 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:00.055+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:01.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:00 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:00.744+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:51:01.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:00 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:00.744+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:51:01.041 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:00 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:00.744+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:30.704083+0000 front 2024-04-26T21:50:30.703944+0000 (oldest deadline 2024-04-26T21:50:53.004036+0000) 2024-04-26T21:51:01.367 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:51:01 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:51:01.130+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:10.359322+0000 front 2024-04-26T21:50:10.359302+0000 (oldest deadline 2024-04-26T21:50:38.560225+0000) 2024-04-26T21:51:01.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:51:01 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:51:01.130+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:21.361781+0000 front 2024-04-26T21:50:21.361918+0000 (oldest deadline 2024-04-26T21:50:49.562305+0000) 2024-04-26T21:51:01.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:51:01 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:51:01.130+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.562999+0000 front 2024-04-26T21:50:29.562945+0000 (oldest deadline 2024-04-26T21:50:54.262883+0000) 2024-04-26T21:51:01.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:00 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:00.972+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:01.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:00 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:00.972+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:01.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:00 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:00.972+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:01.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:00 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:00.972+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:01.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:01 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:01.101+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:01.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:01 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:01.101+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:01.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:01 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:01.101+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:01.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:01 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:01.101+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:01.552 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:01 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:01.207+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:51:01.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:01 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:01.207+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:51:01.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:01 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:01.207+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:51:01.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:01 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:01.207+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:51:02.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:01 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:01.760+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:51:02.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:01 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:01.760+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:51:02.041 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:01 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:01.760+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:30.704083+0000 front 2024-04-26T21:50:30.703944+0000 (oldest deadline 2024-04-26T21:50:53.004036+0000) 2024-04-26T21:51:02.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:51:02 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:51:02.109+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:10.359322+0000 front 2024-04-26T21:50:10.359302+0000 (oldest deadline 2024-04-26T21:50:38.560225+0000) 2024-04-26T21:51:02.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:51:02 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:51:02.109+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:21.361781+0000 front 2024-04-26T21:50:21.361918+0000 (oldest deadline 2024-04-26T21:50:49.562305+0000) 2024-04-26T21:51:02.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:51:02 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:51:02.109+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.562999+0000 front 2024-04-26T21:50:29.562945+0000 (oldest deadline 2024-04-26T21:50:54.262883+0000) 2024-04-26T21:51:02.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:01 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:01.982+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:02.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:01 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:01.982+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:02.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:01 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:01.982+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:02.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:01 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:01.982+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:02.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:02 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:02.123+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:02.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:02 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:02.123+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:02.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:02 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:02.123+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:02.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:02 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:02.123+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:02.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:02 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:02.123+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:02.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:02 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:02.221+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:51:02.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:02 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:02.221+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:51:02.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:02 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:02.221+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:51:02.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:02 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:02.221+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:51:03.162 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:02 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:02.792+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:51:03.163 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:02 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:02.792+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:51:03.163 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:02 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:02.792+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:30.704083+0000 front 2024-04-26T21:50:30.703944+0000 (oldest deadline 2024-04-26T21:50:53.004036+0000) 2024-04-26T21:51:03.163 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:02 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:02.792+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.106278+0000 front 2024-04-26T21:50:38.106306+0000 (oldest deadline 2024-04-26T21:51:02.106786+0000) 2024-04-26T21:51:03.367 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:51:03 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:51:03.087+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:10.359322+0000 front 2024-04-26T21:50:10.359302+0000 (oldest deadline 2024-04-26T21:50:38.560225+0000) 2024-04-26T21:51:03.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:51:03 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:51:03.087+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:21.361781+0000 front 2024-04-26T21:50:21.361918+0000 (oldest deadline 2024-04-26T21:50:49.562305+0000) 2024-04-26T21:51:03.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:51:03 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:51:03.087+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.562999+0000 front 2024-04-26T21:50:29.562945+0000 (oldest deadline 2024-04-26T21:50:54.262883+0000) 2024-04-26T21:51:03.368 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:51:03 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7[50874]: 2024-04-26T21:51:03.087+0000 7fb1e5b1e700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.363951+0000 front 2024-04-26T21:50:38.363992+0000 (oldest deadline 2024-04-26T21:51:02.464033+0000) 2024-04-26T21:51:03.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:03 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:03.017+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:03.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:03 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:03.017+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:03.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:03 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:03.017+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:03.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:03 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:03.017+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:03.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:03 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:03.163+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:03.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:03 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:03.163+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:03.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:03 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:03.163+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:03.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:03 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:03.163+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:03.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:03 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:03.163+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:03.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:03 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:03.254+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:51:03.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:03 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:03.254+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:51:03.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:03 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:03.254+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:51:03.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:03 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:03.254+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:51:04.209 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:03 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:03.826+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:51:04.209 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:03 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:03.826+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:51:04.209 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:03 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:03.826+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:30.704083+0000 front 2024-04-26T21:50:30.703944+0000 (oldest deadline 2024-04-26T21:50:53.004036+0000) 2024-04-26T21:51:04.209 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:03 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:03.826+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.106278+0000 front 2024-04-26T21:50:38.106306+0000 (oldest deadline 2024-04-26T21:51:02.106786+0000) 2024-04-26T21:51:04.322 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:03 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:03.977+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:04.322 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:03 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:03.977+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:04.323 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:03 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:03.977+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:04.323 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:03 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:03.977+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:04.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:04 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:04.208+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:04.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:04 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:04.208+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:04.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:04 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:04.208+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:04.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:04 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:04.208+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:04.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:04 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:04.208+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:04.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:04 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:04.257+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:51:04.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:04 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:04.257+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:51:04.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:04 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:04.257+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:51:04.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:04 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:04.257+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:51:04.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:04 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:04.257+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.398713+0000 front 2024-04-26T21:50:38.398689+0000 (oldest deadline 2024-04-26T21:51:04.199313+0000) 2024-04-26T21:51:04.618 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:51:04 smithi050 podman[68037]: 2024-04-26 21:51:04.322728168 +0000 UTC m=+5.439724465 container died f6a60f9bc7767baa4d38e5e9578cb5d6171f66d8d4ea2aba1aa0280675461d1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5) 2024-04-26T21:51:04.991 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:51:04 smithi050 podman[68037]: 2024-04-26 21:51:04.659464091 +0000 UTC m=+5.776460393 container cleanup f6a60f9bc7767baa4d38e5e9578cb5d6171f66d8d4ea2aba1aa0280675461d1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-26T21:51:04.991 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:51:04 smithi050 bash[68037]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7 2024-04-26T21:51:04.991 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:51:04 smithi050 podman[68055]: 2024-04-26 21:51:04.860771812 +0000 UTC m=+0.532708135 container remove f6a60f9bc7767baa4d38e5e9578cb5d6171f66d8d4ea2aba1aa0280675461d1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-26T21:51:05.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:04 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:04.777+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:51:05.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:04 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:04.777+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:51:05.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:04 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:04.777+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:30.704083+0000 front 2024-04-26T21:50:30.703944+0000 (oldest deadline 2024-04-26T21:50:53.004036+0000) 2024-04-26T21:51:05.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:04 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:04.777+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.106278+0000 front 2024-04-26T21:50:38.106306+0000 (oldest deadline 2024-04-26T21:51:02.106786+0000) 2024-04-26T21:51:05.283 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:04 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:04.990+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:05.283 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:04 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:04.990+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:05.283 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:04 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:04.990+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:05.283 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:04 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:04.990+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:05.283 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:04 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:04.990+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:39.300826+0000 front 2024-04-26T21:50:39.300869+0000 (oldest deadline 2024-04-26T21:51:04.000831+0000) 2024-04-26T21:51:05.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:05 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:05.246+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:05.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:05 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:05.246+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:05.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:05 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:05.246+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:05.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:05 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:05.246+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:05.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:05 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:05.246+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:05.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:05 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:05.301+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:51:05.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:05 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:05.301+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:51:05.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:05 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:05.301+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:51:05.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:05 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:05.301+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:51:05.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:05 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:05.301+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.398713+0000 front 2024-04-26T21:50:38.398689+0000 (oldest deadline 2024-04-26T21:51:04.199313+0000) 2024-04-26T21:51:05.580 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:51:05 smithi050 podman[68146]: 2024-04-26 21:51:05.186752915 +0000 UTC m=+0.018081191 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:51:05.581 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:51:05 smithi050 podman[68146]: 2024-04-26 21:51:05.30339266 +0000 UTC m=+0.134720925 container create 77fec22d5b5baa22bb9ec68fe0526ec8d11331c963e9419095748012e93b8c09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-26T21:51:05.868 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:51:05 smithi050 podman[68146]: 2024-04-26 21:51:05.581362542 +0000 UTC m=+0.412690805 container init 77fec22d5b5baa22bb9ec68fe0526ec8d11331c963e9419095748012e93b8c09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-26T21:51:05.868 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:51:05 smithi050 podman[68146]: 2024-04-26 21:51:05.585045839 +0000 UTC m=+0.416374102 container start 77fec22d5b5baa22bb9ec68fe0526ec8d11331c963e9419095748012e93b8c09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T21:51:05.868 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:51:05 smithi050 podman[68146]: 2024-04-26 21:51:05.71616094 +0000 UTC m=+0.547489217 container attach 77fec22d5b5baa22bb9ec68fe0526ec8d11331c963e9419095748012e93b8c09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:51:06.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:05 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:05.741+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:51:06.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:05 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:05.741+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:51:06.041 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:05 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:05.741+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:30.704083+0000 front 2024-04-26T21:50:30.703944+0000 (oldest deadline 2024-04-26T21:50:53.004036+0000) 2024-04-26T21:51:06.041 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:05 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:05.741+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.106278+0000 front 2024-04-26T21:50:38.106306+0000 (oldest deadline 2024-04-26T21:51:02.106786+0000) 2024-04-26T21:51:06.251 INFO:journalctl@ceph.osd.7.smithi050.stdout:Apr 26 21:51:05 smithi050 podman[68146]: 2024-04-26 21:51:05.888360786 +0000 UTC m=+0.719689042 container died 77fec22d5b5baa22bb9ec68fe0526ec8d11331c963e9419095748012e93b8c09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-26T21:51:06.251 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:05 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:05.949+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:06.251 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:05 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:05.949+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:06.252 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:05 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:05.949+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:06.252 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:05 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:05.949+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:06.252 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:05 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:05.949+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:39.300826+0000 front 2024-04-26T21:50:39.300869+0000 (oldest deadline 2024-04-26T21:51:04.000831+0000) 2024-04-26T21:51:06.684 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.7.service' 2024-04-26T21:51:06.703 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:06 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:06.291+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:06.703 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:06 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:06.291+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:06.703 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:06 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:06.291+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:06.703 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:06 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:06.291+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:06.704 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:06 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:06.291+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:06.730 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:51:06.731 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-26T21:51:06.731 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-26T21:51:06.731 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.8 2024-04-26T21:51:06.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:06 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:06.321+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:51:06.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:06 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:06.321+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:51:06.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:06 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:06.321+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:51:06.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:06 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:06.321+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:51:06.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:06 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:06.321+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.398713+0000 front 2024-04-26T21:50:38.398689+0000 (oldest deadline 2024-04-26T21:51:04.199313+0000) 2024-04-26T21:51:07.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:06 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:06.703+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:51:07.041 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:06 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:06.703+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:51:07.041 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:06 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:06.703+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:30.704083+0000 front 2024-04-26T21:50:30.703944+0000 (oldest deadline 2024-04-26T21:50:53.004036+0000) 2024-04-26T21:51:07.041 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:06 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:06.703+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.106278+0000 front 2024-04-26T21:50:38.106306+0000 (oldest deadline 2024-04-26T21:51:02.106786+0000) 2024-04-26T21:51:07.041 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:06 smithi142 systemd[1]: Stopping Ceph osd.8 for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:51:07.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:06 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:06.993+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:07.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:06 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:06.993+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:07.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:06 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:06.993+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:07.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:06 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:06.993+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:07.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:06 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:06.993+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:39.300826+0000 front 2024-04-26T21:50:39.300869+0000 (oldest deadline 2024-04-26T21:51:04.000831+0000) 2024-04-26T21:51:07.540 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:07 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:07.043+0000 7fbe7d381700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:51:07.540 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:07 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:07.043+0000 7fbe7d381700 -1 osd.8 90 *** Got signal Terminated *** 2024-04-26T21:51:07.541 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:07 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:07.043+0000 7fbe7d381700 -1 osd.8 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T21:51:07.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:07 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:07.258+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:07.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:07 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:07.258+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:07.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:07 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:07.258+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:07.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:07 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:07.258+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:07.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:07 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:07.258+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:07.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:07 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:07.345+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:51:07.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:07 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:07.345+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:51:07.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:07 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:07.345+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:51:07.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:07 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:07.345+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:51:07.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:07 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:07.345+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.398713+0000 front 2024-04-26T21:50:38.398689+0000 (oldest deadline 2024-04-26T21:51:04.199313+0000) 2024-04-26T21:51:08.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:07 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:07.726+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:51:08.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:07 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:07.726+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:51:08.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:07 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:07.726+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:30.704083+0000 front 2024-04-26T21:50:30.703944+0000 (oldest deadline 2024-04-26T21:50:53.004036+0000) 2024-04-26T21:51:08.041 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:07 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:07.726+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.106278+0000 front 2024-04-26T21:50:38.106306+0000 (oldest deadline 2024-04-26T21:51:02.106786+0000) 2024-04-26T21:51:08.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:07 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:07.977+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:08.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:07 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:07.977+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:08.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:07 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:07.977+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:08.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:07 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:07.977+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:08.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:07 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:07.977+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:39.300826+0000 front 2024-04-26T21:50:39.300869+0000 (oldest deadline 2024-04-26T21:51:04.000831+0000) 2024-04-26T21:51:08.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:08 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:08.273+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:08.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:08 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:08.273+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:08.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:08 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:08.273+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:08.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:08 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:08.273+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:08.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:08 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:08.273+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:08.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:08 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:08.333+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:51:08.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:08 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:08.333+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:51:08.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:08 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:08.333+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:51:08.804 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:08 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:08.333+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:51:08.804 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:08 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:08.333+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.398713+0000 front 2024-04-26T21:50:38.398689+0000 (oldest deadline 2024-04-26T21:51:04.199313+0000) 2024-04-26T21:51:09.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:08 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:08.735+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:51:09.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:08 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:08.735+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:51:09.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:08 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:08.735+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:30.704083+0000 front 2024-04-26T21:50:30.703944+0000 (oldest deadline 2024-04-26T21:50:53.004036+0000) 2024-04-26T21:51:09.041 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:08 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:08.735+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.106278+0000 front 2024-04-26T21:50:38.106306+0000 (oldest deadline 2024-04-26T21:51:02.106786+0000) 2024-04-26T21:51:09.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:09 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:09.012+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:09.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:09 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:09.012+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:09.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:09 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:09.012+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:09.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:09 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:09.012+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:09.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:09 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:09.012+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:39.300826+0000 front 2024-04-26T21:50:39.300869+0000 (oldest deadline 2024-04-26T21:51:04.000831+0000) 2024-04-26T21:51:09.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:09 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:09.012+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.001374+0000 front 2024-04-26T21:50:44.001420+0000 (oldest deadline 2024-04-26T21:51:08.701340+0000) 2024-04-26T21:51:09.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:09 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:09.271+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:09.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:09 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:09.271+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:09.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:09 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:09.271+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:09.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:09 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:09.271+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:09.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:09 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:09.271+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:09.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:09 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:09.294+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:51:09.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:09 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:09.294+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:51:09.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:09 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:09.294+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:51:09.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:09 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:09.294+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:51:09.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:09 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:09.294+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.398713+0000 front 2024-04-26T21:50:38.398689+0000 (oldest deadline 2024-04-26T21:51:04.199313+0000) 2024-04-26T21:51:10.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:09 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:09.739+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:51:10.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:09 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:09.739+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:51:10.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:09 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:09.739+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:30.704083+0000 front 2024-04-26T21:50:30.703944+0000 (oldest deadline 2024-04-26T21:50:53.004036+0000) 2024-04-26T21:51:10.041 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:09 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:09.739+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.106278+0000 front 2024-04-26T21:50:38.106306+0000 (oldest deadline 2024-04-26T21:51:02.106786+0000) 2024-04-26T21:51:10.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:09 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:09.965+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:10.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:09 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:09.965+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:10.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:09 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:09.965+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:10.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:09 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:09.965+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:10.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:09 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:09.965+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:39.300826+0000 front 2024-04-26T21:50:39.300869+0000 (oldest deadline 2024-04-26T21:51:04.000831+0000) 2024-04-26T21:51:10.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:09 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:09.965+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.001374+0000 front 2024-04-26T21:50:44.001420+0000 (oldest deadline 2024-04-26T21:51:08.701340+0000) 2024-04-26T21:51:10.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:10 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:10.232+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:10.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:10 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:10.232+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:10.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:10 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:10.232+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:10.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:10 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:10.232+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:10.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:10 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:10.232+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:10.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:10 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:10.268+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:51:10.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:10 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:10.268+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:51:10.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:10 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:10.268+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:51:10.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:10 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:10.268+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:51:10.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:10 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:10.268+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.398713+0000 front 2024-04-26T21:50:38.398689+0000 (oldest deadline 2024-04-26T21:51:04.199313+0000) 2024-04-26T21:51:11.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:10 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:10.782+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:51:11.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:10 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:10.782+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:51:11.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:10 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:10.782+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:30.704083+0000 front 2024-04-26T21:50:30.703944+0000 (oldest deadline 2024-04-26T21:50:53.004036+0000) 2024-04-26T21:51:11.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:10 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:10.782+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.106278+0000 front 2024-04-26T21:50:38.106306+0000 (oldest deadline 2024-04-26T21:51:02.106786+0000) 2024-04-26T21:51:11.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:10 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:10.987+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:11.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:10 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:10.987+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:11.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:10 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:10.987+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:11.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:10 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:10.987+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:11.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:10 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:10.987+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:39.300826+0000 front 2024-04-26T21:50:39.300869+0000 (oldest deadline 2024-04-26T21:51:04.000831+0000) 2024-04-26T21:51:11.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:10 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:10.987+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.001374+0000 front 2024-04-26T21:50:44.001420+0000 (oldest deadline 2024-04-26T21:51:08.701340+0000) 2024-04-26T21:51:11.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:11 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:11.202+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:11.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:11 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:11.202+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:11.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:11 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:11.202+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:11.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:11 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:11.202+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:11.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:11 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:11.202+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:11.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:11 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:11.202+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.410006+0000 front 2024-04-26T21:50:44.409893+0000 (oldest deadline 2024-04-26T21:51:10.309954+0000) 2024-04-26T21:51:11.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:11 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:11.318+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:51:11.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:11 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:11.318+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:51:11.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:11 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:11.318+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:51:11.804 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:11 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:11.318+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:51:11.804 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:11 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:11.318+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.398713+0000 front 2024-04-26T21:50:38.398689+0000 (oldest deadline 2024-04-26T21:51:04.199313+0000) 2024-04-26T21:51:12.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:11 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:11.771+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.100222+0000 front 2024-04-26T21:50:14.100310+0000 (oldest deadline 2024-04-26T21:50:38.800039+0000) 2024-04-26T21:51:12.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:11 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:11.771+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:19.300834+0000 front 2024-04-26T21:50:19.300854+0000 (oldest deadline 2024-04-26T21:50:45.201018+0000) 2024-04-26T21:51:12.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:11 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:11.771+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:30.704083+0000 front 2024-04-26T21:50:30.703944+0000 (oldest deadline 2024-04-26T21:50:53.004036+0000) 2024-04-26T21:51:12.041 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:11 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:11.771+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.106278+0000 front 2024-04-26T21:50:38.106306+0000 (oldest deadline 2024-04-26T21:51:02.106786+0000) 2024-04-26T21:51:12.041 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:11 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8[51842]: 2024-04-26T21:51:11.771+0000 7fbe7596d700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:47.308071+0000 front 2024-04-26T21:50:47.308089+0000 (oldest deadline 2024-04-26T21:51:11.308533+0000) 2024-04-26T21:51:12.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:11 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:11.996+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:12.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:11 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:11.996+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:12.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:11 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:11.996+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:12.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:11 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:11.996+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:12.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:11 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:11.996+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:39.300826+0000 front 2024-04-26T21:50:39.300869+0000 (oldest deadline 2024-04-26T21:51:04.000831+0000) 2024-04-26T21:51:12.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:11 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:11.996+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.001374+0000 front 2024-04-26T21:50:44.001420+0000 (oldest deadline 2024-04-26T21:51:08.701340+0000) 2024-04-26T21:51:12.472 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:12 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:12.169+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:12.472 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:12 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:12.169+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:12.472 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:12 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:12.169+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:12.472 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:12 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:12.169+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:12.472 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:12 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:12.169+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:12.472 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:12 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:12.169+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.410006+0000 front 2024-04-26T21:50:44.409893+0000 (oldest deadline 2024-04-26T21:51:10.309954+0000) 2024-04-26T21:51:12.728 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:12 smithi142 podman[69199]: 2024-04-26 21:51:12.471495807 +0000 UTC m=+5.547247965 container died 5363d0630b2e9ff3c9993764777daa5da2d8319c97b50bd59e3d80a8655f6d91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-26T21:51:12.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:12 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:12.364+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:51:12.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:12 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:12.365+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:51:12.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:12 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:12.365+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:51:12.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:12 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:12.365+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:51:12.804 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:12 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:12.365+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.398713+0000 front 2024-04-26T21:50:38.398689+0000 (oldest deadline 2024-04-26T21:51:04.199313+0000) 2024-04-26T21:51:12.804 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:12 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:12.365+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:45.900154+0000 front 2024-04-26T21:50:45.900184+0000 (oldest deadline 2024-04-26T21:51:11.700519+0000) 2024-04-26T21:51:13.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:12 smithi142 podman[69199]: 2024-04-26 21:51:12.82836981 +0000 UTC m=+5.904121960 container cleanup 5363d0630b2e9ff3c9993764777daa5da2d8319c97b50bd59e3d80a8655f6d91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:51:13.040 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:12 smithi142 bash[69199]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8 2024-04-26T21:51:13.367 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:12 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:12.989+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:13.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:12 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:12.989+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:13.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:12 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:12.989+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:13.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:12 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:12.989+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:13.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:12 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:12.989+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:39.300826+0000 front 2024-04-26T21:50:39.300869+0000 (oldest deadline 2024-04-26T21:51:04.000831+0000) 2024-04-26T21:51:13.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:12 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:12.989+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.001374+0000 front 2024-04-26T21:50:44.001420+0000 (oldest deadline 2024-04-26T21:51:08.701340+0000) 2024-04-26T21:51:13.449 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:13 smithi142 podman[69216]: 2024-04-26 21:51:13.121369514 +0000 UTC m=+0.644545723 container remove 5363d0630b2e9ff3c9993764777daa5da2d8319c97b50bd59e3d80a8655f6d91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T21:51:13.449 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:13 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:13.186+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:13.450 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:13 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:13.186+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:13.450 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:13 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:13.186+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:13.450 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:13 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:13.186+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:13.450 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:13 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:13.186+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:13.450 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:13 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:13.186+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.410006+0000 front 2024-04-26T21:50:44.409893+0000 (oldest deadline 2024-04-26T21:51:10.309954+0000) 2024-04-26T21:51:13.702 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:13 smithi142 podman[69302]: 2024-04-26 21:51:13.352434106 +0000 UTC m=+0.018106297 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:51:13.703 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:13 smithi142 podman[69302]: 2024-04-26 21:51:13.471988347 +0000 UTC m=+0.137660523 container create 16beaa7380bde98e07c23ba1b07a73fb5b0576460546f6db1b80e85ec7d5b355 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-26T21:51:13.703 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:13 smithi142 podman[69302]: 2024-04-26 21:51:13.699753759 +0000 UTC m=+0.365425933 container init 16beaa7380bde98e07c23ba1b07a73fb5b0576460546f6db1b80e85ec7d5b355 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-26T21:51:13.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:13 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:13.349+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:51:13.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:13 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:13.349+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:51:13.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:13 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:13.349+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:51:13.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:13 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:13.349+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:51:13.804 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:13 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:13.349+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.398713+0000 front 2024-04-26T21:50:38.398689+0000 (oldest deadline 2024-04-26T21:51:04.199313+0000) 2024-04-26T21:51:13.804 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:13 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:13.349+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:45.900154+0000 front 2024-04-26T21:50:45.900184+0000 (oldest deadline 2024-04-26T21:51:11.700519+0000) 2024-04-26T21:51:13.995 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:13 smithi142 podman[69302]: 2024-04-26 21:51:13.702958511 +0000 UTC m=+0.368630685 container start 16beaa7380bde98e07c23ba1b07a73fb5b0576460546f6db1b80e85ec7d5b355 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-26T21:51:13.996 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:13 smithi142 podman[69302]: 2024-04-26 21:51:13.799758141 +0000 UTC m=+0.465430328 container attach 16beaa7380bde98e07c23ba1b07a73fb5b0576460546f6db1b80e85ec7d5b355 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=) 2024-04-26T21:51:14.271 INFO:journalctl@ceph.osd.8.smithi142.stdout:Apr 26 21:51:13 smithi142 podman[69302]: 2024-04-26 21:51:13.996136777 +0000 UTC m=+0.661808943 container died 16beaa7380bde98e07c23ba1b07a73fb5b0576460546f6db1b80e85ec7d5b355 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-8-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-26T21:51:14.271 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:14 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:14.189+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:14.271 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:14 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:14.189+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:14.271 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:14 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:14.189+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:14.271 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:14 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:14.189+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:14.271 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:14 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:14.189+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:14.272 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:14 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:14.189+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.410006+0000 front 2024-04-26T21:50:44.409893+0000 (oldest deadline 2024-04-26T21:51:10.309954+0000) 2024-04-26T21:51:14.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:13 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:13.974+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:14.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:13 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:13.974+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:14.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:13 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:13.974+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:14.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:13 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:13.974+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:14.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:13 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:13.974+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:39.300826+0000 front 2024-04-26T21:50:39.300869+0000 (oldest deadline 2024-04-26T21:51:04.000831+0000) 2024-04-26T21:51:14.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:13 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:13.974+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.001374+0000 front 2024-04-26T21:50:44.001420+0000 (oldest deadline 2024-04-26T21:51:08.701340+0000) 2024-04-26T21:51:14.713 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.8.service' 2024-04-26T21:51:14.750 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:51:14.751 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-26T21:51:14.751 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-26T21:51:14.751 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.9 2024-04-26T21:51:14.780 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:14 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:14.345+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:51:14.780 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:14 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:14.345+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:51:14.780 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:14 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:14.345+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:51:14.780 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:14 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:14.345+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:51:14.780 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:14 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:14.345+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.398713+0000 front 2024-04-26T21:50:38.398689+0000 (oldest deadline 2024-04-26T21:51:04.199313+0000) 2024-04-26T21:51:14.781 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:14 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:14.345+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:45.900154+0000 front 2024-04-26T21:50:45.900184+0000 (oldest deadline 2024-04-26T21:51:11.700519+0000) 2024-04-26T21:51:15.053 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:14 smithi046 systemd[1]: Stopping Ceph osd.9 for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:51:15.335 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:15 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:15.071+0000 7f25ddce7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:51:15.335 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:15 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:15.071+0000 7f25ddce7700 -1 osd.9 90 *** Got signal Terminated *** 2024-04-26T21:51:15.335 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:15 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:15.071+0000 7f25ddce7700 -1 osd.9 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T21:51:15.336 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:15 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:15.334+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:51:15.336 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:15 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:15.334+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:51:15.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:14 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:14.943+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:15.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:14 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:14.943+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:15.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:14 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:14.943+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:15.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:14 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:14.943+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:15.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:14 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:14.943+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:39.300826+0000 front 2024-04-26T21:50:39.300869+0000 (oldest deadline 2024-04-26T21:51:04.000831+0000) 2024-04-26T21:51:15.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:14 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:14.943+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.001374+0000 front 2024-04-26T21:50:44.001420+0000 (oldest deadline 2024-04-26T21:51:08.701340+0000) 2024-04-26T21:51:15.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:15 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:15.199+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:15.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:15 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:15.199+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:15.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:15 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:15.199+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:15.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:15 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:15.199+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:15.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:15 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:15.199+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:15.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:15 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:15.199+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.410006+0000 front 2024-04-26T21:50:44.409893+0000 (oldest deadline 2024-04-26T21:51:10.309954+0000) 2024-04-26T21:51:15.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:15 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:15.334+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:51:15.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:15 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:15.334+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:51:15.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:15 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:15.334+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.398713+0000 front 2024-04-26T21:50:38.398689+0000 (oldest deadline 2024-04-26T21:51:04.199313+0000) 2024-04-26T21:51:15.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:15 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:15.334+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:45.900154+0000 front 2024-04-26T21:50:45.900184+0000 (oldest deadline 2024-04-26T21:51:11.700519+0000) 2024-04-26T21:51:16.269 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:15 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:15.992+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:16.269 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:15 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:15.992+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:16.269 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:15 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:15.992+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:16.270 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:15 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:15.992+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:16.270 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:15 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:15.992+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:39.300826+0000 front 2024-04-26T21:50:39.300869+0000 (oldest deadline 2024-04-26T21:51:04.000831+0000) 2024-04-26T21:51:16.270 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:15 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:15.992+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.001374+0000 front 2024-04-26T21:50:44.001420+0000 (oldest deadline 2024-04-26T21:51:08.701340+0000) 2024-04-26T21:51:16.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:16 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:16.225+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:16.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:16 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:16.225+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:16.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:16 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:16.225+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:16.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:16 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:16.225+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:16.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:16 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:16.225+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:16.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:16 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:16.225+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.410006+0000 front 2024-04-26T21:50:44.409893+0000 (oldest deadline 2024-04-26T21:51:10.309954+0000) 2024-04-26T21:51:16.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:16 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:16.291+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:51:16.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:16 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:16.291+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:51:16.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:16 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:16.291+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:51:16.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:16 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:16.291+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:51:16.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:16 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:16.291+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.398713+0000 front 2024-04-26T21:50:38.398689+0000 (oldest deadline 2024-04-26T21:51:04.199313+0000) 2024-04-26T21:51:16.553 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:16 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:16.291+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:45.900154+0000 front 2024-04-26T21:50:45.900184+0000 (oldest deadline 2024-04-26T21:51:11.700519+0000) 2024-04-26T21:51:17.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:16 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:16.977+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:17.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:16 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:16.977+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:17.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:16 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:16.977+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:17.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:16 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:16.977+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:17.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:16 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:16.977+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:39.300826+0000 front 2024-04-26T21:50:39.300869+0000 (oldest deadline 2024-04-26T21:51:04.000831+0000) 2024-04-26T21:51:17.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:16 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:16.977+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.001374+0000 front 2024-04-26T21:50:44.001420+0000 (oldest deadline 2024-04-26T21:51:08.701340+0000) 2024-04-26T21:51:17.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:17 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:17.211+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:17.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:17 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:17.211+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:17.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:17 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:17.211+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:17.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:17 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:17.211+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:17.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:17 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:17.211+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:17.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:17 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:17.211+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.410006+0000 front 2024-04-26T21:50:44.409893+0000 (oldest deadline 2024-04-26T21:51:10.309954+0000) 2024-04-26T21:51:17.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:17 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:17.330+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:51:17.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:17 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:17.330+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:51:17.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:17 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:17.330+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:51:17.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:17 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:17.330+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:51:17.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:17 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:17.330+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.398713+0000 front 2024-04-26T21:50:38.398689+0000 (oldest deadline 2024-04-26T21:51:04.199313+0000) 2024-04-26T21:51:17.804 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:17 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:17.330+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:45.900154+0000 front 2024-04-26T21:50:45.900184+0000 (oldest deadline 2024-04-26T21:51:11.700519+0000) 2024-04-26T21:51:18.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:18 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:18.020+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:18.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:18 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:18.020+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:18.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:18 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:18.020+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:18.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:18 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:18.020+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:18.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:18 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:18.020+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:39.300826+0000 front 2024-04-26T21:50:39.300869+0000 (oldest deadline 2024-04-26T21:51:04.000831+0000) 2024-04-26T21:51:18.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:18 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:18.020+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.001374+0000 front 2024-04-26T21:50:44.001420+0000 (oldest deadline 2024-04-26T21:51:08.701340+0000) 2024-04-26T21:51:18.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:18 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:18.248+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:18.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:18 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:18.248+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:18.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:18 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:18.248+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:18.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:18 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:18.248+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:18.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:18 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:18.248+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:18.542 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:18 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:18.248+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.410006+0000 front 2024-04-26T21:50:44.409893+0000 (oldest deadline 2024-04-26T21:51:10.309954+0000) 2024-04-26T21:51:18.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:18 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:18.333+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:51:18.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:18 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:18.333+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:51:18.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:18 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:18.333+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:51:18.804 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:18 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:18.333+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:51:18.804 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:18 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:18.333+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.398713+0000 front 2024-04-26T21:50:38.398689+0000 (oldest deadline 2024-04-26T21:51:04.199313+0000) 2024-04-26T21:51:18.804 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:18 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:18.333+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:45.900154+0000 front 2024-04-26T21:50:45.900184+0000 (oldest deadline 2024-04-26T21:51:11.700519+0000) 2024-04-26T21:51:19.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:19 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:19.009+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:19.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:19 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:19.009+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:19.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:19 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:19.009+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:19.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:19 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:19.009+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:19.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:19 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:19.009+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:39.300826+0000 front 2024-04-26T21:50:39.300869+0000 (oldest deadline 2024-04-26T21:51:04.000831+0000) 2024-04-26T21:51:19.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:19 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:19.009+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.001374+0000 front 2024-04-26T21:50:44.001420+0000 (oldest deadline 2024-04-26T21:51:08.701340+0000) 2024-04-26T21:51:19.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:19 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:19.263+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:19.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:19 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:19.263+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:19.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:19 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:19.263+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:19.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:19 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:19.263+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:19.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:19 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:19.263+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:19.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:19 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:19.263+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.410006+0000 front 2024-04-26T21:50:44.409893+0000 (oldest deadline 2024-04-26T21:51:10.309954+0000) 2024-04-26T21:51:19.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:19 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:19.321+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.993210+0000 front 2024-04-26T21:50:06.993175+0000 (oldest deadline 2024-04-26T21:50:28.093313+0000) 2024-04-26T21:51:19.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:19 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:19.321+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:13.894694+0000 front 2024-04-26T21:50:13.894636+0000 (oldest deadline 2024-04-26T21:50:39.194831+0000) 2024-04-26T21:51:19.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:19 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:19.321+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:23.796627+0000 front 2024-04-26T21:50:23.796524+0000 (oldest deadline 2024-04-26T21:50:44.896500+0000) 2024-04-26T21:51:19.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:19 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:19.321+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:28.997425+0000 front 2024-04-26T21:50:28.997494+0000 (oldest deadline 2024-04-26T21:50:52.497539+0000) 2024-04-26T21:51:19.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:19 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:19.321+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:38.398713+0000 front 2024-04-26T21:50:38.398689+0000 (oldest deadline 2024-04-26T21:51:04.199313+0000) 2024-04-26T21:51:19.804 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:19 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9[63910]: 2024-04-26T21:51:19.321+0000 7f25da2db700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:45.900154+0000 front 2024-04-26T21:50:45.900184+0000 (oldest deadline 2024-04-26T21:51:11.700519+0000) 2024-04-26T21:51:20.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:19 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:19.976+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:20.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:19 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:19.976+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:20.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:19 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:19.976+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:20.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:19 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:19.976+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:20.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:19 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:19.976+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:39.300826+0000 front 2024-04-26T21:50:39.300869+0000 (oldest deadline 2024-04-26T21:51:04.000831+0000) 2024-04-26T21:51:20.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:19 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:19.976+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.001374+0000 front 2024-04-26T21:50:44.001420+0000 (oldest deadline 2024-04-26T21:51:08.701340+0000) 2024-04-26T21:51:20.790 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:20 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:20.312+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:20.791 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:20 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:20.312+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:20.791 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:20 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:20.312+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:20.791 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:20 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:20.312+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:20.791 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:20 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:20.312+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:20.791 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:20 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:20.312+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.410006+0000 front 2024-04-26T21:50:44.409893+0000 (oldest deadline 2024-04-26T21:51:10.309954+0000) 2024-04-26T21:51:20.916 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:20 smithi046 podman[85475]: 2024-04-26 21:51:20.58539289 +0000 UTC m=+5.635774330 container died 9f43b5373b15efb8a73a71951c7025ff947b24969b95194c8e81016c1829133e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T21:51:21.230 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:20 smithi046 podman[85475]: 2024-04-26 21:51:20.996909279 +0000 UTC m=+6.047290728 container cleanup 9f43b5373b15efb8a73a71951c7025ff947b24969b95194c8e81016c1829133e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-26T21:51:21.231 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:20 smithi046 bash[85475]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9 2024-04-26T21:51:21.367 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:20 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:20.967+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:21.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:20 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:20.967+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:21.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:20 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:20.967+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:21.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:20 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:20.967+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:21.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:20 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:20.967+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:39.300826+0000 front 2024-04-26T21:50:39.300869+0000 (oldest deadline 2024-04-26T21:51:04.000831+0000) 2024-04-26T21:51:21.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:20 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:20.967+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.001374+0000 front 2024-04-26T21:50:44.001420+0000 (oldest deadline 2024-04-26T21:51:08.701340+0000) 2024-04-26T21:51:21.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:20 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:20.967+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-04-26T21:50:54.602630+0000 front 2024-04-26T21:50:54.602504+0000 (oldest deadline 2024-04-26T21:51:20.402811+0000) 2024-04-26T21:51:21.535 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:21 smithi046 podman[85492]: 2024-04-26 21:51:21.231180064 +0000 UTC m=+0.636099564 container remove 9f43b5373b15efb8a73a71951c7025ff947b24969b95194c8e81016c1829133e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:51:21.790 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:21 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:21.319+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:21.790 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:21 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:21.319+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:21.790 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:21 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:21.319+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:21.791 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:21 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:21.319+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:21.791 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:21 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:21.319+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:21.791 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:21 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:21.319+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.410006+0000 front 2024-04-26T21:50:44.409893+0000 (oldest deadline 2024-04-26T21:51:10.309954+0000) 2024-04-26T21:51:21.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:21 smithi046 podman[85585]: 2024-04-26 21:51:21.439250548 +0000 UTC m=+0.018075749 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:51:21.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:21 smithi046 podman[85585]: 2024-04-26 21:51:21.540122984 +0000 UTC m=+0.118948171 container create 2a6b9956b45689e31aaee3339eca933779ce6fdd892f5587022deca78fc8d28b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:51:21.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:21 smithi046 podman[85585]: 2024-04-26 21:51:21.76728968 +0000 UTC m=+0.346114867 container init 2a6b9956b45689e31aaee3339eca933779ce6fdd892f5587022deca78fc8d28b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T21:51:21.803 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:21 smithi046 podman[85585]: 2024-04-26 21:51:21.770483527 +0000 UTC m=+0.349308715 container start 2a6b9956b45689e31aaee3339eca933779ce6fdd892f5587022deca78fc8d28b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-26T21:51:22.078 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:21 smithi046 podman[85585]: 2024-04-26 21:51:21.875633787 +0000 UTC m=+0.454458976 container attach 2a6b9956b45689e31aaee3339eca933779ce6fdd892f5587022deca78fc8d28b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-26T21:51:22.361 INFO:journalctl@ceph.osd.9.smithi046.stdout:Apr 26 21:51:22 smithi046 podman[85585]: 2024-04-26 21:51:22.078800127 +0000 UTC m=+0.657625324 container died 2a6b9956b45689e31aaee3339eca933779ce6fdd892f5587022deca78fc8d28b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-9-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T21:51:22.367 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:21 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:21.992+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:22.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:21 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:21.992+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:22.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:21 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:21.992+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:22.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:21 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:21.992+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:22.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:21 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:21.992+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:39.300826+0000 front 2024-04-26T21:50:39.300869+0000 (oldest deadline 2024-04-26T21:51:04.000831+0000) 2024-04-26T21:51:22.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:21 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:21.992+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.001374+0000 front 2024-04-26T21:50:44.001420+0000 (oldest deadline 2024-04-26T21:51:08.701340+0000) 2024-04-26T21:51:22.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:21 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:21.992+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-04-26T21:50:54.602630+0000 front 2024-04-26T21:50:54.602504+0000 (oldest deadline 2024-04-26T21:51:20.402811+0000) 2024-04-26T21:51:22.696 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.9.service' 2024-04-26T21:51:22.776 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:51:22.776 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-26T21:51:22.777 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-26T21:51:22.777 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.10 2024-04-26T21:51:22.790 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:22 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:22.317+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:22.790 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:22 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:22.317+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:22.790 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:22 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:22.317+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:22.790 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:22 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:22.317+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:22.791 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:22 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:22.317+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:22.791 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:22 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:22.317+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.410006+0000 front 2024-04-26T21:50:44.409893+0000 (oldest deadline 2024-04-26T21:51:10.309954+0000) 2024-04-26T21:51:23.118 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:22 smithi050 systemd[1]: Stopping Ceph osd.10 for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:51:23.118 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:22 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:22.945+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:23.118 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:22 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:22.945+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:23.119 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:22 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:22.945+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:23.119 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:22 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:22.945+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:23.119 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:22 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:22.945+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:39.300826+0000 front 2024-04-26T21:50:39.300869+0000 (oldest deadline 2024-04-26T21:51:04.000831+0000) 2024-04-26T21:51:23.119 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:22 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:22.945+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.001374+0000 front 2024-04-26T21:50:44.001420+0000 (oldest deadline 2024-04-26T21:51:08.701340+0000) 2024-04-26T21:51:23.119 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:22 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:22.945+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-04-26T21:50:54.602630+0000 front 2024-04-26T21:50:54.602504+0000 (oldest deadline 2024-04-26T21:51:20.402811+0000) 2024-04-26T21:51:23.618 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:23 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:23.229+0000 7fae99a0e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:51:23.618 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:23 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:23.229+0000 7fae99a0e700 -1 osd.10 90 *** Got signal Terminated *** 2024-04-26T21:51:23.618 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:23 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:23.229+0000 7fae99a0e700 -1 osd.10 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T21:51:23.790 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:23 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:23.332+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:23.791 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:23 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:23.332+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:23.791 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:23 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:23.332+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:23.791 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:23 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:23.332+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:23.791 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:23 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:23.332+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:23.791 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:23 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:23.332+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.410006+0000 front 2024-04-26T21:50:44.409893+0000 (oldest deadline 2024-04-26T21:51:10.309954+0000) 2024-04-26T21:51:24.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:23 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:23.973+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:24.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:23 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:23.973+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:24.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:23 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:23.973+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:24.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:23 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:23.973+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:24.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:23 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:23.973+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:39.300826+0000 front 2024-04-26T21:50:39.300869+0000 (oldest deadline 2024-04-26T21:51:04.000831+0000) 2024-04-26T21:51:24.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:23 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:23.973+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.001374+0000 front 2024-04-26T21:50:44.001420+0000 (oldest deadline 2024-04-26T21:51:08.701340+0000) 2024-04-26T21:51:24.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:23 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:23.973+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-04-26T21:50:54.602630+0000 front 2024-04-26T21:50:54.602504+0000 (oldest deadline 2024-04-26T21:51:20.402811+0000) 2024-04-26T21:51:24.790 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:24 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:24.349+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:24.791 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:24 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:24.349+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:24.791 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:24 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:24.349+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:24.791 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:24 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:24.349+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:24.791 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:24 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:24.349+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:24.791 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:24 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:24.349+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.410006+0000 front 2024-04-26T21:50:44.409893+0000 (oldest deadline 2024-04-26T21:51:10.309954+0000) 2024-04-26T21:51:25.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:24 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:24.956+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:25.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:24 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:24.956+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:25.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:24 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:24.956+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:25.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:24 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:24.956+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:25.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:24 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:24.956+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:39.300826+0000 front 2024-04-26T21:50:39.300869+0000 (oldest deadline 2024-04-26T21:51:04.000831+0000) 2024-04-26T21:51:25.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:24 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:24.956+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.001374+0000 front 2024-04-26T21:50:44.001420+0000 (oldest deadline 2024-04-26T21:51:08.701340+0000) 2024-04-26T21:51:25.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:24 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:24.956+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-04-26T21:50:54.602630+0000 front 2024-04-26T21:50:54.602504+0000 (oldest deadline 2024-04-26T21:51:20.402811+0000) 2024-04-26T21:51:25.790 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:25 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:25.308+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:25.790 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:25 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:25.308+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:25.791 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:25 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:25.308+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:25.791 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:25 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:25.308+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:25.791 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:25 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:25.308+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:25.791 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:25 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:25.308+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.410006+0000 front 2024-04-26T21:50:44.409893+0000 (oldest deadline 2024-04-26T21:51:10.309954+0000) 2024-04-26T21:51:26.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:25 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:25.946+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:26.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:25 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:25.946+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:26.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:25 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:25.946+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:26.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:25 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:25.946+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:26.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:25 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:25.946+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:39.300826+0000 front 2024-04-26T21:50:39.300869+0000 (oldest deadline 2024-04-26T21:51:04.000831+0000) 2024-04-26T21:51:26.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:25 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:25.946+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.001374+0000 front 2024-04-26T21:50:44.001420+0000 (oldest deadline 2024-04-26T21:51:08.701340+0000) 2024-04-26T21:51:26.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:25 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:25.946+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-04-26T21:50:54.602630+0000 front 2024-04-26T21:50:54.602504+0000 (oldest deadline 2024-04-26T21:51:20.402811+0000) 2024-04-26T21:51:26.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:26 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:26.261+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:26.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:26 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:26.261+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:26.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:26 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:26.261+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:26.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:26 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:26.261+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:26.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:26 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:26.261+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:26.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:26 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:26.261+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.410006+0000 front 2024-04-26T21:50:44.409893+0000 (oldest deadline 2024-04-26T21:51:10.309954+0000) 2024-04-26T21:51:27.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:26 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:26.974+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:27.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:26 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:26.974+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:27.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:26 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:26.974+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:27.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:26 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:26.974+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:27.368 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:26 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:26.974+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:39.300826+0000 front 2024-04-26T21:50:39.300869+0000 (oldest deadline 2024-04-26T21:51:04.000831+0000) 2024-04-26T21:51:27.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:26 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:26.974+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.001374+0000 front 2024-04-26T21:50:44.001420+0000 (oldest deadline 2024-04-26T21:51:08.701340+0000) 2024-04-26T21:51:27.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:26 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:26.974+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-04-26T21:50:54.602630+0000 front 2024-04-26T21:50:54.602504+0000 (oldest deadline 2024-04-26T21:51:20.402811+0000) 2024-04-26T21:51:27.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:27 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:27.243+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:27.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:27 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:27.243+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:27.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:27 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:27.243+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:27.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:27 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:27.243+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:27.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:27 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:27.243+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:27.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:27 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:27.243+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.410006+0000 front 2024-04-26T21:50:44.409893+0000 (oldest deadline 2024-04-26T21:51:10.309954+0000) 2024-04-26T21:51:27.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:27 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:27.243+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6820 osd.7 since back 2024-04-26T21:51:00.112677+0000 front 2024-04-26T21:51:00.112664+0000 (oldest deadline 2024-04-26T21:51:27.112897+0000) 2024-04-26T21:51:28.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:27 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:27.960+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.695238+0000 front 2024-04-26T21:50:06.695470+0000 (oldest deadline 2024-04-26T21:50:28.395319+0000) 2024-04-26T21:51:28.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:27 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:27.960+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:14.797103+0000 front 2024-04-26T21:50:14.797229+0000 (oldest deadline 2024-04-26T21:50:39.497232+0000) 2024-04-26T21:51:28.369 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:27 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:27.960+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:22.299073+0000 front 2024-04-26T21:50:22.298978+0000 (oldest deadline 2024-04-26T21:50:48.198978+0000) 2024-04-26T21:51:28.370 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:27 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:27.960+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:31.100025+0000 front 2024-04-26T21:50:31.100096+0000 (oldest deadline 2024-04-26T21:50:55.799868+0000) 2024-04-26T21:51:28.370 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:27 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:27.960+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:39.300826+0000 front 2024-04-26T21:50:39.300869+0000 (oldest deadline 2024-04-26T21:51:04.000831+0000) 2024-04-26T21:51:28.370 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:27 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:27.960+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.001374+0000 front 2024-04-26T21:50:44.001420+0000 (oldest deadline 2024-04-26T21:51:08.701340+0000) 2024-04-26T21:51:28.370 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:27 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10[57664]: 2024-04-26T21:51:27.960+0000 7fae91ffa700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-04-26T21:50:54.602630+0000 front 2024-04-26T21:50:54.602504+0000 (oldest deadline 2024-04-26T21:51:20.402811+0000) 2024-04-26T21:51:28.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:28 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:28.211+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:28.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:28 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:28.211+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:28.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:28 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:28.211+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:28.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:28 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:28.211+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:28.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:28 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:28.211+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:28.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:28 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:28.211+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.410006+0000 front 2024-04-26T21:50:44.409893+0000 (oldest deadline 2024-04-26T21:51:10.309954+0000) 2024-04-26T21:51:28.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:28 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:28.211+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6820 osd.7 since back 2024-04-26T21:51:00.112677+0000 front 2024-04-26T21:51:00.112664+0000 (oldest deadline 2024-04-26T21:51:27.112897+0000) 2024-04-26T21:51:28.846 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:28 smithi050 podman[68265]: 2024-04-26 21:51:28.577314076 +0000 UTC m=+5.594494409 container died 5ddb6d50850f31d813ed19b72827a41f621216be8f25d569a66b7c0657918019 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-26T21:51:29.118 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:28 smithi050 podman[68265]: 2024-04-26 21:51:28.937854137 +0000 UTC m=+5.955034463 container cleanup 5ddb6d50850f31d813ed19b72827a41f621216be8f25d569a66b7c0657918019 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-26T21:51:29.119 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:28 smithi050 bash[68265]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10 2024-04-26T21:51:29.519 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:29 smithi050 podman[68280]: 2024-04-26 21:51:29.205648621 +0000 UTC m=+0.623783422 container remove 5ddb6d50850f31d813ed19b72827a41f621216be8f25d569a66b7c0657918019 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424) 2024-04-26T21:51:29.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:29 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:29.210+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:29.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:29 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:29.210+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:29.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:29 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:29.210+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:29.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:29 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:29.210+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:29.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:29 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:29.210+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:29.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:29 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:29.210+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.410006+0000 front 2024-04-26T21:50:44.409893+0000 (oldest deadline 2024-04-26T21:51:10.309954+0000) 2024-04-26T21:51:29.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:29 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:29.210+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6820 osd.7 since back 2024-04-26T21:51:00.112677+0000 front 2024-04-26T21:51:00.112664+0000 (oldest deadline 2024-04-26T21:51:27.112897+0000) 2024-04-26T21:51:29.868 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:29 smithi050 podman[68376]: 2024-04-26 21:51:29.422698553 +0000 UTC m=+0.017918167 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:51:29.868 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:29 smithi050 podman[68376]: 2024-04-26 21:51:29.639746446 +0000 UTC m=+0.234966049 container create 407f12f46e2c361f6a0c55c9d31a76dc1e1f9f8c6fa19aa26da5cff2946cc88e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10-deactivate, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-26T21:51:30.181 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:29 smithi050 podman[68376]: 2024-04-26 21:51:29.875617275 +0000 UTC m=+0.470836879 container init 407f12f46e2c361f6a0c55c9d31a76dc1e1f9f8c6fa19aa26da5cff2946cc88e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-26T21:51:30.181 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:29 smithi050 podman[68376]: 2024-04-26 21:51:29.878766771 +0000 UTC m=+0.473986375 container start 407f12f46e2c361f6a0c55c9d31a76dc1e1f9f8c6fa19aa26da5cff2946cc88e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-26T21:51:30.181 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:29 smithi050 podman[68376]: 2024-04-26 21:51:29.950405444 +0000 UTC m=+0.545625057 container attach 407f12f46e2c361f6a0c55c9d31a76dc1e1f9f8c6fa19aa26da5cff2946cc88e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-26T21:51:30.461 INFO:journalctl@ceph.osd.10.smithi050.stdout:Apr 26 21:51:30 smithi050 podman[68376]: 2024-04-26 21:51:30.181581771 +0000 UTC m=+0.776801374 container died 407f12f46e2c361f6a0c55c9d31a76dc1e1f9f8c6fa19aa26da5cff2946cc88e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-10-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-26T21:51:30.504 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:30 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:30.235+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:30.504 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:30 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:30.235+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:30.504 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:30 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:30.235+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:30.504 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:30 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:30.235+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:30.504 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:30 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:30.235+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:30.505 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:30 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:30.235+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.410006+0000 front 2024-04-26T21:50:44.409893+0000 (oldest deadline 2024-04-26T21:51:10.309954+0000) 2024-04-26T21:51:30.505 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:30 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:30.235+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6820 osd.7 since back 2024-04-26T21:51:00.112677+0000 front 2024-04-26T21:51:00.112664+0000 (oldest deadline 2024-04-26T21:51:27.112897+0000) 2024-04-26T21:51:30.747 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.10.service' 2024-04-26T21:51:30.788 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:51:30.788 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-26T21:51:30.789 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-26T21:51:30.789 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.11 2024-04-26T21:51:31.072 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:30 smithi142 systemd[1]: Stopping Ceph osd.11 for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:51:31.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:31 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:31.072+0000 7fc80a40d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:51:31.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:31 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:31.072+0000 7fc80a40d700 -1 osd.11 90 *** Got signal Terminated *** 2024-04-26T21:51:31.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:31 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:31.072+0000 7fc80a40d700 -1 osd.11 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T21:51:31.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:31 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:31.271+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:31.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:31 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:31.271+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:31.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:31 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:31.271+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:31.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:31 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:31.271+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:31.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:31 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:31.271+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:31.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:31 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:31.271+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.410006+0000 front 2024-04-26T21:50:44.409893+0000 (oldest deadline 2024-04-26T21:51:10.309954+0000) 2024-04-26T21:51:31.542 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:31 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:31.271+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6820 osd.7 since back 2024-04-26T21:51:00.112677+0000 front 2024-04-26T21:51:00.112664+0000 (oldest deadline 2024-04-26T21:51:27.112897+0000) 2024-04-26T21:51:32.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:32 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:32.289+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:32.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:32 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:32.289+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:32.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:32 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:32.289+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:32.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:32 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:32.289+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:32.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:32 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:32.289+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:32.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:32 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:32.289+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.410006+0000 front 2024-04-26T21:50:44.409893+0000 (oldest deadline 2024-04-26T21:51:10.309954+0000) 2024-04-26T21:51:32.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:32 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:32.289+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6820 osd.7 since back 2024-04-26T21:51:00.112677+0000 front 2024-04-26T21:51:00.112664+0000 (oldest deadline 2024-04-26T21:51:27.112897+0000) 2024-04-26T21:51:33.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:33 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:33.268+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:33.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:33 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:33.268+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:33.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:33 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:33.268+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:33.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:33 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:33.268+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:33.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:33 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:33.268+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:33.542 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:33 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:33.268+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.410006+0000 front 2024-04-26T21:50:44.409893+0000 (oldest deadline 2024-04-26T21:51:10.309954+0000) 2024-04-26T21:51:33.542 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:33 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:33.268+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6820 osd.7 since back 2024-04-26T21:51:00.112677+0000 front 2024-04-26T21:51:00.112664+0000 (oldest deadline 2024-04-26T21:51:27.112897+0000) 2024-04-26T21:51:34.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:34 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:34.251+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:34.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:34 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:34.251+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:34.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:34 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:34.251+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:34.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:34 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:34.251+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:34.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:34 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:34.251+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:34.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:34 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:34.251+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.410006+0000 front 2024-04-26T21:50:44.409893+0000 (oldest deadline 2024-04-26T21:51:10.309954+0000) 2024-04-26T21:51:34.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:34 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:34.251+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6820 osd.7 since back 2024-04-26T21:51:00.112677+0000 front 2024-04-26T21:51:00.112664+0000 (oldest deadline 2024-04-26T21:51:27.112897+0000) 2024-04-26T21:51:35.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:35 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:35.256+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-04-26T21:50:06.504666+0000 front 2024-04-26T21:50:06.504502+0000 (oldest deadline 2024-04-26T21:50:31.204613+0000) 2024-04-26T21:51:35.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:35 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:35.256+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6804 osd.1 since back 2024-04-26T21:50:12.806434+0000 front 2024-04-26T21:50:12.806308+0000 (oldest deadline 2024-04-26T21:50:38.706342+0000) 2024-04-26T21:51:35.540 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:35 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:35.256+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6804 osd.2 since back 2024-04-26T21:50:18.707148+0000 front 2024-04-26T21:50:18.706721+0000 (oldest deadline 2024-04-26T21:50:49.307291+0000) 2024-04-26T21:51:35.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:35 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:35.256+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-04-26T21:50:29.808106+0000 front 2024-04-26T21:50:29.808118+0000 (oldest deadline 2024-04-26T21:50:55.108070+0000) 2024-04-26T21:51:35.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:35 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:35.256+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6812 osd.4 since back 2024-04-26T21:50:35.108678+0000 front 2024-04-26T21:50:35.108718+0000 (oldest deadline 2024-04-26T21:51:02.109265+0000) 2024-04-26T21:51:35.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:35 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:35.256+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6812 osd.5 since back 2024-04-26T21:50:44.410006+0000 front 2024-04-26T21:50:44.409893+0000 (oldest deadline 2024-04-26T21:51:10.309954+0000) 2024-04-26T21:51:35.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:35 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:35.256+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.50:6820 osd.7 since back 2024-04-26T21:51:00.112677+0000 front 2024-04-26T21:51:00.112664+0000 (oldest deadline 2024-04-26T21:51:27.112897+0000) 2024-04-26T21:51:35.541 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:35 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11[58883]: 2024-04-26T21:51:35.256+0000 7fc8029f9700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.142:6820 osd.8 since back 2024-04-26T21:51:07.113221+0000 front 2024-04-26T21:51:07.113246+0000 (oldest deadline 2024-04-26T21:51:34.713748+0000) 2024-04-26T21:51:36.713 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:36 smithi142 podman[69420]: 2024-04-26 21:51:36.396344049 +0000 UTC m=+5.420613369 container died 64d973f436ec7baf655717ce03be2fb31a05d186eb309d438cf9e0e97ced96ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-26T21:51:36.713 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:36 smithi142 podman[69420]: 2024-04-26 21:51:36.71365653 +0000 UTC m=+5.737925841 container cleanup 64d973f436ec7baf655717ce03be2fb31a05d186eb309d438cf9e0e97ced96ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T21:51:37.040 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:36 smithi142 bash[69420]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11 2024-04-26T21:51:37.040 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:36 smithi142 podman[69437]: 2024-04-26 21:51:36.914512783 +0000 UTC m=+0.515475505 container remove 64d973f436ec7baf655717ce03be2fb31a05d186eb309d438cf9e0e97ced96ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-26T21:51:37.626 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:37 smithi142 podman[69529]: 2024-04-26 21:51:37.331985167 +0000 UTC m=+0.113901632 container create 29ffa1fd83cd50dc3bb0da0e5e8f9a84feb8858eae4eb93bd3c5398a153969cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-26T21:51:37.627 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:37 smithi142 podman[69529]: 2024-04-26 21:51:37.23630359 +0000 UTC m=+0.018220044 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:51:37.627 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:37 smithi142 podman[69529]: 2024-04-26 21:51:37.551713352 +0000 UTC m=+0.333629787 container init 29ffa1fd83cd50dc3bb0da0e5e8f9a84feb8858eae4eb93bd3c5398a153969cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:51:37.627 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:37 smithi142 podman[69529]: 2024-04-26 21:51:37.554807177 +0000 UTC m=+0.336723603 container start 29ffa1fd83cd50dc3bb0da0e5e8f9a84feb8858eae4eb93bd3c5398a153969cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-26T21:51:38.040 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:37 smithi142 podman[69529]: 2024-04-26 21:51:37.626647998 +0000 UTC m=+0.408564431 container attach 29ffa1fd83cd50dc3bb0da0e5e8f9a84feb8858eae4eb93bd3c5398a153969cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-26T21:51:38.041 INFO:journalctl@ceph.osd.11.smithi142.stdout:Apr 26 21:51:37 smithi142 podman[69529]: 2024-04-26 21:51:37.847507641 +0000 UTC m=+0.629424078 container died 29ffa1fd83cd50dc3bb0da0e5e8f9a84feb8858eae4eb93bd3c5398a153969cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T21:51:38.406 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@osd.11.service' 2024-04-26T21:51:38.443 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:51:38.443 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-26T21:51:38.443 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-26T21:51:38.443 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.a 2024-04-26T21:51:38.763 INFO:journalctl@ceph.mds.a.smithi046.stdout:Apr 26 21:51:38 smithi046 systemd[1]: Stopping Ceph mds.a for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:51:39.053 INFO:journalctl@ceph.mds.a.smithi046.stdout:Apr 26 21:51:38 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-a[70611]: 2024-04-26T21:51:38.763+0000 7f3a5d0ec700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:51:39.053 INFO:journalctl@ceph.mds.a.smithi046.stdout:Apr 26 21:51:38 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-a[70611]: 2024-04-26T21:51:38.763+0000 7f3a5d0ec700 -1 mds.a *** got signal Terminated *** 2024-04-26T21:51:40.478 INFO:journalctl@ceph.mds.a.smithi046.stdout:Apr 26 21:51:40 smithi046 podman[85708]: 2024-04-26 21:51:40.055949009 +0000 UTC m=+1.413318418 container died 28ef6aefc492b0a2aebe8d10781d281f7ec3123620077d55ded6c72f7a42505c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-a, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T21:51:40.795 INFO:journalctl@ceph.mds.a.smithi046.stdout:Apr 26 21:51:40 smithi046 podman[85708]: 2024-04-26 21:51:40.578101087 +0000 UTC m=+1.935470503 container cleanup 28ef6aefc492b0a2aebe8d10781d281f7ec3123620077d55ded6c72f7a42505c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-26T21:51:40.795 INFO:journalctl@ceph.mds.a.smithi046.stdout:Apr 26 21:51:40 smithi046 bash[85708]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-a 2024-04-26T21:51:41.009 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.a.service' 2024-04-26T21:51:41.043 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:51:41.043 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-26T21:51:41.043 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-26T21:51:41.044 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.d 2024-04-26T21:51:41.365 INFO:journalctl@ceph.mds.d.smithi046.stdout:Apr 26 21:51:41 smithi046 systemd[1]: Stopping Ceph mds.d for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:51:41.803 INFO:journalctl@ceph.mds.d.smithi046.stdout:Apr 26 21:51:41 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-d[69638]: 2024-04-26T21:51:41.364+0000 7fc33cfe8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:51:41.803 INFO:journalctl@ceph.mds.d.smithi046.stdout:Apr 26 21:51:41 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-d[69638]: 2024-04-26T21:51:41.364+0000 7fc33cfe8700 -1 mds.d *** got signal Terminated *** 2024-04-26T21:51:44.053 INFO:journalctl@ceph.mds.d.smithi046.stdout:Apr 26 21:51:43 smithi046 podman[85830]: 2024-04-26 21:51:43.827442303 +0000 UTC m=+2.566974567 container died 04bfdfb737279183a375a429450ac4d60d0309f68e45461702311c7a0b181f0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-d, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True) 2024-04-26T21:51:44.553 INFO:journalctl@ceph.mds.d.smithi046.stdout:Apr 26 21:51:44 smithi046 podman[85830]: 2024-04-26 21:51:44.306687 +0000 UTC m=+3.046219251 container cleanup 04bfdfb737279183a375a429450ac4d60d0309f68e45461702311c7a0b181f0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-d, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T21:51:44.553 INFO:journalctl@ceph.mds.d.smithi046.stdout:Apr 26 21:51:44 smithi046 bash[85830]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-d 2024-04-26T21:51:44.787 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.d.service' 2024-04-26T21:51:44.821 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:51:44.821 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-26T21:51:44.821 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-26T21:51:44.821 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.g 2024-04-26T21:51:45.053 INFO:journalctl@ceph.mds.g.smithi046.stdout:Apr 26 21:51:44 smithi046 systemd[1]: Stopping Ceph mds.g for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:51:45.553 INFO:journalctl@ceph.mds.g.smithi046.stdout:Apr 26 21:51:45 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-g[70244]: 2024-04-26T21:51:45.134+0000 7fc310ef4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:51:45.553 INFO:journalctl@ceph.mds.g.smithi046.stdout:Apr 26 21:51:45 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-g[70244]: 2024-04-26T21:51:45.134+0000 7fc310ef4700 -1 mds.g *** got signal Terminated *** 2024-04-26T21:51:49.344 INFO:journalctl@ceph.mds.g.smithi046.stdout:Apr 26 21:51:49 smithi046 podman[85951]: 2024-04-26 21:51:49.055776271 +0000 UTC m=+4.017462759 container died d4aed68ced68dbf421a1d2678c48c5e269f5f9b4b8d862d0ad6d786e1d7bf4a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-g, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-26T21:51:49.702 INFO:journalctl@ceph.mds.g.smithi046.stdout:Apr 26 21:51:49 smithi046 podman[85951]: 2024-04-26 21:51:49.426695787 +0000 UTC m=+4.388382281 container cleanup d4aed68ced68dbf421a1d2678c48c5e269f5f9b4b8d862d0ad6d786e1d7bf4a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-26T21:51:49.702 INFO:journalctl@ceph.mds.g.smithi046.stdout:Apr 26 21:51:49 smithi046 bash[85951]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-g 2024-04-26T21:51:49.866 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.g.service' 2024-04-26T21:51:49.898 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:51:49.898 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-26T21:51:49.898 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-26T21:51:49.898 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.j 2024-04-26T21:51:50.213 INFO:journalctl@ceph.mds.j.smithi046.stdout:Apr 26 21:51:49 smithi046 systemd[1]: Stopping Ceph mds.j for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:51:50.553 INFO:journalctl@ceph.mds.j.smithi046.stdout:Apr 26 21:51:50 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-j[69292]: 2024-04-26T21:51:50.212+0000 7f3b375d0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:51:50.553 INFO:journalctl@ceph.mds.j.smithi046.stdout:Apr 26 21:51:50 smithi046 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-j[69292]: 2024-04-26T21:51:50.212+0000 7f3b375d0700 -1 mds.j *** got signal Terminated *** 2024-04-26T21:51:53.096 INFO:journalctl@ceph.mds.j.smithi046.stdout:Apr 26 21:51:52 smithi046 podman[86072]: 2024-04-26 21:51:52.821291558 +0000 UTC m=+2.703003816 container died a38069bfbd621c83a40728775bea4129db3882e1d8b404031acdc0c0ee630d7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-j, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True) 2024-04-26T21:51:53.413 INFO:journalctl@ceph.mds.j.smithi046.stdout:Apr 26 21:51:53 smithi046 podman[86072]: 2024-04-26 21:51:53.196539377 +0000 UTC m=+3.078251637 container cleanup a38069bfbd621c83a40728775bea4129db3882e1d8b404031acdc0c0ee630d7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-j, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-26T21:51:53.414 INFO:journalctl@ceph.mds.j.smithi046.stdout:Apr 26 21:51:53 smithi046 bash[86072]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-j 2024-04-26T21:51:53.565 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.j.service' 2024-04-26T21:51:53.596 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:51:53.596 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-26T21:51:53.596 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-26T21:51:53.596 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.b 2024-04-26T21:51:53.913 INFO:journalctl@ceph.mds.b.smithi050.stdout:Apr 26 21:51:53 smithi050 systemd[1]: Stopping Ceph mds.b for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:51:54.368 INFO:journalctl@ceph.mds.b.smithi050.stdout:Apr 26 21:51:53 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-b[63049]: 2024-04-26T21:51:53.913+0000 7f292aa46700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:51:54.368 INFO:journalctl@ceph.mds.b.smithi050.stdout:Apr 26 21:51:53 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-b[63049]: 2024-04-26T21:51:53.913+0000 7f292aa46700 -1 mds.b *** got signal Terminated *** 2024-04-26T21:51:58.188 INFO:journalctl@ceph.mds.b.smithi050.stdout:Apr 26 21:51:57 smithi050 podman[68482]: 2024-04-26 21:51:57.928422517 +0000 UTC m=+4.131970061 container died f0907047d44e3ba7d721dd63ac9a0079adc3d03d0fbff2f02e8c74e3afaf5ca1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-b, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5) 2024-04-26T21:51:58.450 INFO:journalctl@ceph.mds.b.smithi050.stdout:Apr 26 21:51:58 smithi050 podman[68482]: 2024-04-26 21:51:58.262676621 +0000 UTC m=+4.466224157 container cleanup f0907047d44e3ba7d721dd63ac9a0079adc3d03d0fbff2f02e8c74e3afaf5ca1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-26T21:51:58.450 INFO:journalctl@ceph.mds.b.smithi050.stdout:Apr 26 21:51:58 smithi050 bash[68482]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-b 2024-04-26T21:51:58.673 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.b.service' 2024-04-26T21:51:58.709 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:51:58.709 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-26T21:51:58.710 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-26T21:51:58.710 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.e 2024-04-26T21:51:59.066 INFO:journalctl@ceph.mds.e.smithi050.stdout:Apr 26 21:51:58 smithi050 systemd[1]: Stopping Ceph mds.e for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:51:59.368 INFO:journalctl@ceph.mds.e.smithi050.stdout:Apr 26 21:51:59 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-e[62428]: 2024-04-26T21:51:59.066+0000 7fbf242d8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:51:59.368 INFO:journalctl@ceph.mds.e.smithi050.stdout:Apr 26 21:51:59 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-e[62428]: 2024-04-26T21:51:59.066+0000 7fbf242d8700 -1 mds.e *** got signal Terminated *** 2024-04-26T21:52:03.094 INFO:journalctl@ceph.mds.e.smithi050.stdout:Apr 26 21:52:02 smithi050 podman[68611]: 2024-04-26 21:52:02.752971208 +0000 UTC m=+3.810672241 container died 9ba3ee1dbc35cc537aa87040c86086277ad612a7f262735ce1c37d9a231ee806 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-e, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:52:03.368 INFO:journalctl@ceph.mds.e.smithi050.stdout:Apr 26 21:52:03 smithi050 podman[68611]: 2024-04-26 21:52:03.20812191 +0000 UTC m=+4.265822932 container cleanup 9ba3ee1dbc35cc537aa87040c86086277ad612a7f262735ce1c37d9a231ee806 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-e, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-26T21:52:03.368 INFO:journalctl@ceph.mds.e.smithi050.stdout:Apr 26 21:52:03 smithi050 bash[68611]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-e 2024-04-26T21:52:03.606 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.e.service' 2024-04-26T21:52:03.640 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:52:03.641 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-26T21:52:03.641 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-26T21:52:03.641 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.h 2024-04-26T21:52:03.867 INFO:journalctl@ceph.mds.h.smithi050.stdout:Apr 26 21:52:03 smithi050 systemd[1]: Stopping Ceph mds.h for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:52:04.367 INFO:journalctl@ceph.mds.h.smithi050.stdout:Apr 26 21:52:03 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-h[62739]: 2024-04-26T21:52:03.945+0000 7f3c63d44700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:52:04.368 INFO:journalctl@ceph.mds.h.smithi050.stdout:Apr 26 21:52:03 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-h[62739]: 2024-04-26T21:52:03.945+0000 7f3c63d44700 -1 mds.h *** got signal Terminated *** 2024-04-26T21:52:07.021 INFO:journalctl@ceph.mds.h.smithi050.stdout:Apr 26 21:52:06 smithi050 podman[68732]: 2024-04-26 21:52:06.76708962 +0000 UTC m=+2.906159843 container died 84e5daa6aa3301d06da29da5be8a926ab8a0fa4d8f229de78639a8ad32e32a19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-h, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424) 2024-04-26T21:52:07.368 INFO:journalctl@ceph.mds.h.smithi050.stdout:Apr 26 21:52:07 smithi050 podman[68732]: 2024-04-26 21:52:07.111885405 +0000 UTC m=+3.250955628 container cleanup 84e5daa6aa3301d06da29da5be8a926ab8a0fa4d8f229de78639a8ad32e32a19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True) 2024-04-26T21:52:07.368 INFO:journalctl@ceph.mds.h.smithi050.stdout:Apr 26 21:52:07 smithi050 bash[68732]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-h 2024-04-26T21:52:07.537 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.h.service' 2024-04-26T21:52:07.569 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:52:07.570 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-26T21:52:07.570 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-26T21:52:07.570 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.k 2024-04-26T21:52:07.898 INFO:journalctl@ceph.mds.k.smithi050.stdout:Apr 26 21:52:07 smithi050 systemd[1]: Stopping Ceph mds.k for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:52:08.368 INFO:journalctl@ceph.mds.k.smithi050.stdout:Apr 26 21:52:07 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-k[62109]: 2024-04-26T21:52:07.897+0000 7fea9bfc8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:52:08.368 INFO:journalctl@ceph.mds.k.smithi050.stdout:Apr 26 21:52:07 smithi050 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-k[62109]: 2024-04-26T21:52:07.897+0000 7fea9bfc8700 -1 mds.k *** got signal Terminated *** 2024-04-26T21:52:10.107 INFO:journalctl@ceph.mds.k.smithi050.stdout:Apr 26 21:52:09 smithi050 podman[68853]: 2024-04-26 21:52:09.829446467 +0000 UTC m=+2.034716680 container died 07c58350ace453e95076c10bdf87c5589591dfc0b76322b588b173db9e231cfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-k, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T21:52:10.107 INFO:journalctl@ceph.mds.k.smithi050.stdout:Apr 26 21:52:10 smithi050 podman[68853]: 2024-04-26 21:52:10.107411682 +0000 UTC m=+2.312681912 container cleanup 07c58350ace453e95076c10bdf87c5589591dfc0b76322b588b173db9e231cfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-k, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-26T21:52:10.367 INFO:journalctl@ceph.mds.k.smithi050.stdout:Apr 26 21:52:10 smithi050 bash[68853]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-k 2024-04-26T21:52:10.693 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.k.service' 2024-04-26T21:52:10.725 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:52:10.726 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-26T21:52:10.726 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-26T21:52:10.726 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.c 2024-04-26T21:52:11.012 INFO:journalctl@ceph.mds.c.smithi142.stdout:Apr 26 21:52:10 smithi142 systemd[1]: Stopping Ceph mds.c for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:52:11.273 INFO:journalctl@ceph.mds.c.smithi142.stdout:Apr 26 21:52:11 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-c[63877]: 2024-04-26T21:52:11.011+0000 7fa793414700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:52:11.273 INFO:journalctl@ceph.mds.c.smithi142.stdout:Apr 26 21:52:11 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-c[63877]: 2024-04-26T21:52:11.011+0000 7fa793414700 -1 mds.c *** got signal Terminated *** 2024-04-26T21:52:15.037 INFO:journalctl@ceph.mds.c.smithi142.stdout:Apr 26 21:52:14 smithi142 podman[69633]: 2024-04-26 21:52:14.750802109 +0000 UTC m=+3.836444554 container died 365169d6c55a96dd3500b05a864caed7338878e6e50b1dc7695cbc75e707cce4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T21:52:15.290 INFO:journalctl@ceph.mds.c.smithi142.stdout:Apr 26 21:52:15 smithi142 podman[69633]: 2024-04-26 21:52:15.128503603 +0000 UTC m=+4.214146088 container cleanup 365169d6c55a96dd3500b05a864caed7338878e6e50b1dc7695cbc75e707cce4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=) 2024-04-26T21:52:15.290 INFO:journalctl@ceph.mds.c.smithi142.stdout:Apr 26 21:52:15 smithi142 bash[69633]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-c 2024-04-26T21:52:15.509 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.c.service' 2024-04-26T21:52:15.543 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:52:15.543 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-26T21:52:15.544 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-26T21:52:15.544 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.f 2024-04-26T21:52:15.790 INFO:journalctl@ceph.mds.f.smithi142.stdout:Apr 26 21:52:15 smithi142 systemd[1]: Stopping Ceph mds.f for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:52:16.290 INFO:journalctl@ceph.mds.f.smithi142.stdout:Apr 26 21:52:15 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-f[63568]: 2024-04-26T21:52:15.890+0000 7f3caa2a7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:52:16.291 INFO:journalctl@ceph.mds.f.smithi142.stdout:Apr 26 21:52:15 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-f[63568]: 2024-04-26T21:52:15.890+0000 7f3caa2a7700 -1 mds.f *** got signal Terminated *** 2024-04-26T21:52:19.040 INFO:journalctl@ceph.mds.f.smithi142.stdout:Apr 26 21:52:18 smithi142 podman[69767]: 2024-04-26 21:52:18.74970159 +0000 UTC m=+2.987064114 container died 2bf98b6eacb3c827841f375600cd8c06bc10a7cb8c711da5b9b2c9bb58b696b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-26T21:52:19.374 INFO:journalctl@ceph.mds.f.smithi142.stdout:Apr 26 21:52:19 smithi142 podman[69767]: 2024-04-26 21:52:19.123848124 +0000 UTC m=+3.361210655 container cleanup 2bf98b6eacb3c827841f375600cd8c06bc10a7cb8c711da5b9b2c9bb58b696b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-26T21:52:19.374 INFO:journalctl@ceph.mds.f.smithi142.stdout:Apr 26 21:52:19 smithi142 bash[69767]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-f 2024-04-26T21:52:19.480 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.f.service' 2024-04-26T21:52:19.512 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:52:19.512 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-26T21:52:19.512 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-26T21:52:19.512 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.i 2024-04-26T21:52:19.790 INFO:journalctl@ceph.mds.i.smithi142.stdout:Apr 26 21:52:19 smithi142 systemd[1]: Stopping Ceph mds.i for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:52:20.290 INFO:journalctl@ceph.mds.i.smithi142.stdout:Apr 26 21:52:19 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-i[64184]: 2024-04-26T21:52:19.835+0000 7f4d450c3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:52:20.290 INFO:journalctl@ceph.mds.i.smithi142.stdout:Apr 26 21:52:19 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-i[64184]: 2024-04-26T21:52:19.835+0000 7f4d450c3700 -1 mds.i *** got signal Terminated *** 2024-04-26T21:52:23.040 INFO:journalctl@ceph.mds.i.smithi142.stdout:Apr 26 21:52:22 smithi142 podman[69887]: 2024-04-26 21:52:22.757036765 +0000 UTC m=+3.031039697 container died 5e031c0f66ac7b93447c42fff00a924770d887629eeac154caca5e4ce8f16715 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-i, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-26T21:52:23.372 INFO:journalctl@ceph.mds.i.smithi142.stdout:Apr 26 21:52:23 smithi142 podman[69887]: 2024-04-26 21:52:23.061014926 +0000 UTC m=+3.335017842 container cleanup 5e031c0f66ac7b93447c42fff00a924770d887629eeac154caca5e4ce8f16715 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-i, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-26T21:52:23.372 INFO:journalctl@ceph.mds.i.smithi142.stdout:Apr 26 21:52:23 smithi142 bash[69887]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-i 2024-04-26T21:52:23.373 INFO:journalctl@ceph.mds.i.smithi142.stdout:Apr 26 21:52:23 smithi142 podman[69901]: 2024-04-26 21:52:23.287159604 +0000 UTC m=+0.529983946 container remove 5e031c0f66ac7b93447c42fff00a924770d887629eeac154caca5e4ce8f16715 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-i, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS) 2024-04-26T21:52:23.383 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.i.service' 2024-04-26T21:52:23.413 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:52:23.414 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-26T21:52:23.414 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-26T21:52:23.414 DEBUG:teuthology.orchestra.run.smithi142:> sudo systemctl stop ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.l 2024-04-26T21:52:23.714 INFO:journalctl@ceph.mds.l.smithi142.stdout:Apr 26 21:52:23 smithi142 systemd[1]: Stopping Ceph mds.l for 36c7013a-0413-11ef-bc93-c7b262605968... 2024-04-26T21:52:24.040 INFO:journalctl@ceph.mds.l.smithi142.stdout:Apr 26 21:52:23 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-l[63259]: 2024-04-26T21:52:23.714+0000 7f83254e1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:52:24.040 INFO:journalctl@ceph.mds.l.smithi142.stdout:Apr 26 21:52:23 smithi142 ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-l[63259]: 2024-04-26T21:52:23.714+0000 7f83254e1700 -1 mds.l *** got signal Terminated *** 2024-04-26T21:52:27.040 INFO:journalctl@ceph.mds.l.smithi142.stdout:Apr 26 21:52:26 smithi142 podman[70010]: 2024-04-26 21:52:26.778954629 +0000 UTC m=+3.147338197 container died de319134d9918817af28adf90c2e8fb065658755b910192868787131f6d6cb3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-l, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-26T21:52:27.540 INFO:journalctl@ceph.mds.l.smithi142.stdout:Apr 26 21:52:27 smithi142 podman[70010]: 2024-04-26 21:52:27.181382501 +0000 UTC m=+3.549766056 container cleanup de319134d9918817af28adf90c2e8fb065658755b910192868787131f6d6cb3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-l, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5) 2024-04-26T21:52:27.540 INFO:journalctl@ceph.mds.l.smithi142.stdout:Apr 26 21:52:27 smithi142 bash[70010]: ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-l 2024-04-26T21:52:27.540 INFO:journalctl@ceph.mds.l.smithi142.stdout:Apr 26 21:52:27 smithi142 podman[70024]: 2024-04-26 21:52:27.35720336 +0000 UTC m=+0.580835928 container remove de319134d9918817af28adf90c2e8fb065658755b910192868787131f6d6cb3f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-36c7013a-0413-11ef-bc93-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-26T21:52:27.603 DEBUG:teuthology.orchestra.run.smithi142:> sudo pkill -f 'journalctl -f -n 0 -u ceph-36c7013a-0413-11ef-bc93-c7b262605968@mds.l.service' 2024-04-26T21:52:27.633 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:52:27.633 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-26T21:52:27.634 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 36c7013a-0413-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T21:52:27.832 INFO:teuthology.orchestra.run.smithi046.stdout:Deleting cluster with fsid: 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:52:30.383 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 36c7013a-0413-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T21:52:30.580 INFO:teuthology.orchestra.run.smithi050.stdout:Deleting cluster with fsid: 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:52:33.142 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 36c7013a-0413-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T21:52:33.331 INFO:teuthology.orchestra.run.smithi142.stdout:Deleting cluster with fsid: 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:52:35.595 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T21:52:35.628 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T21:52:35.661 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T21:52:35.686 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-26T21:52:35.688 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675215/remote/smithi046/crash 2024-04-26T21:52:35.689 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/crash -- . 2024-04-26T21:52:35.723 INFO:teuthology.orchestra.run.smithi046.stderr:tar: /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T21:52:35.723 INFO:teuthology.orchestra.run.smithi046.stderr:tar: Error is not recoverable: exiting now 2024-04-26T21:52:35.725 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675215/remote/smithi050/crash 2024-04-26T21:52:35.726 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/crash -- . 2024-04-26T21:52:35.765 INFO:teuthology.orchestra.run.smithi050.stderr:tar: /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T21:52:35.766 INFO:teuthology.orchestra.run.smithi050.stderr:tar: Error is not recoverable: exiting now 2024-04-26T21:52:35.767 DEBUG:teuthology.misc:Transferring archived files from smithi142:/var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675215/remote/smithi142/crash 2024-04-26T21:52:35.768 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar c -f - -C /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/crash -- . 2024-04-26T21:52:35.805 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /var/lib/ceph/36c7013a-0413-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T21:52:35.805 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now 2024-04-26T21:52:35.806 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-26T21:52:35.806 DEBUG:teuthology.orchestra.run.smithi046:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-26T21:52:35.845 INFO:tasks.cephadm:Compressing logs... 2024-04-26T21:52:35.845 DEBUG:teuthology.orchestra.run.smithi046:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T21:52:35.888 DEBUG:teuthology.orchestra.run.smithi050:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T21:52:35.891 DEBUG:teuthology.orchestra.run.smithi142:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T21:52:35.914 INFO:teuthology.orchestra.run.smithi046.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.log: No such file or directory 2024-04-26T21:52:35.914 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-04-26T21:52:35.915 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T21:52:35.915 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-26T21:52:35.915 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-26T21:52:35.915 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-26T21:52:35.929 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-26T21:52:35.930 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.j.log: /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-26T21:52:35.930 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- 89.1% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T21:52:35.930 INFO:teuthology.orchestra.run.smithi046.stderr: /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-26T21:52:35.930 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-26T21:52:35.930 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T21:52:35.930 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mon.a.log: /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.6.log: 90.3% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T21:52:35.930 INFO:teuthology.orchestra.run.smithi046.stderr: 94.6% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-26T21:52:35.930 INFO:teuthology.orchestra.run.smithi046.stderr: 94.7% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-26T21:52:35.931 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-26T21:52:35.931 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 82.8% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T21:52:35.931 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T21:52:35.931 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-26T21:52:35.931 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-26T21:52:35.931 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.d.log: 94.6% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-26T21:52:35.931 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T21:52:35.931 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.9.log: 94.6% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-26T21:52:35.934 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-26T21:52:35.934 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.log 2024-04-26T21:52:35.934 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-26T21:52:35.934 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-26T21:52:35.934 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T21:52:35.935 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-26T21:52:35.935 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-26T21:52:35.935 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.audit.log: /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-26T21:52:35.935 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-26T21:52:35.935 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-26T21:52:35.935 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mgr.z.log: 89.2% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T21:52:35.935 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T21:52:35.935 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.i.log: 90.5% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T21:52:35.935 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-26T21:52:35.936 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.cephadm.log: 82.5% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T21:52:35.936 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-26T21:52:35.936 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.11.log: 93.2% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-26T21:52:35.936 INFO:teuthology.orchestra.run.smithi142.stderr:gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T21:52:35.937 INFO:teuthology.orchestra.run.smithi050.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.log 2024-04-26T21:52:35.937 INFO:teuthology.orchestra.run.smithi050.stderr:: No such file or directory 2024-04-26T21:52:35.937 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T21:52:35.937 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-26T21:52:35.937 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-26T21:52:35.937 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-26T21:52:35.938 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-26T21:52:35.938 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.b.log: /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-26T21:52:35.938 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-26T21:52:35.938 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-26T21:52:35.938 INFO:teuthology.orchestra.run.smithi050.stderr: 89.2% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T21:52:35.938 INFO:teuthology.orchestra.run.smithi050.stderr: 90.5% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T21:52:35.938 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T21:52:35.938 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-26T21:52:35.939 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.cephadm.log: 82.5% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T21:52:35.939 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T21:52:35.939 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.e.log: 93.1% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-26T21:52:35.939 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/cephadm.log: 91.1% 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T21:52:35.940 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T21:52:35.944 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-26T21:52:35.944 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-26T21:52:35.951 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T21:52:35.956 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-26T21:52:35.958 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mon.b.log: 91.1% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T21:52:35.967 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T21:52:35.970 INFO:teuthology.orchestra.run.smithi142.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.l.log: 94.6%/var/log/ceph/cephadm.log: -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-26T21:52:35.970 INFO:teuthology.orchestra.run.smithi142.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T21:52:35.970 INFO:teuthology.orchestra.run.smithi142.stderr: 91.0% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T21:52:35.980 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.k.log: 94.1% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-26T21:52:35.982 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cephadm.log: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T21:52:35.982 INFO:teuthology.orchestra.run.smithi050.stderr: 94.4% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-26T21:52:35.983 INFO:teuthology.orchestra.run.smithi142.stderr: 93.8% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-26T21:52:35.987 INFO:teuthology.orchestra.run.smithi142.stderr: 94.7% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-26T21:52:35.997 INFO:teuthology.orchestra.run.smithi142.stderr: 94.6% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-26T21:52:36.000 INFO:teuthology.orchestra.run.smithi050.stderr: 94.6% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-26T21:52:36.124 INFO:teuthology.orchestra.run.smithi050.stderr: 93.9% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-26T21:52:36.161 INFO:teuthology.orchestra.run.smithi046.stderr: 90.6% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-26T21:52:36.333 INFO:teuthology.orchestra.run.smithi142.stderr: 92.1% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-26T21:52:36.380 INFO:teuthology.orchestra.run.smithi050.stderr: 92.0% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-26T21:52:36.534 INFO:teuthology.orchestra.run.smithi050.stderr: 92.7% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-26T21:52:36.549 INFO:teuthology.orchestra.run.smithi142.stderr: 92.4% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-26T21:52:36.602 INFO:teuthology.orchestra.run.smithi046.stderr: 92.2% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-26T21:52:36.604 INFO:teuthology.orchestra.run.smithi046.stderr: 92.6% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-26T21:52:36.680 INFO:teuthology.orchestra.run.smithi046.stderr: 92.5% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-26T21:52:36.834 INFO:teuthology.orchestra.run.smithi142.stderr: 93.5% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-26T21:52:36.919 INFO:teuthology.orchestra.run.smithi142.stderr: 93.2% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-26T21:52:37.040 INFO:teuthology.orchestra.run.smithi046.stderr: 90.3% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-26T21:52:37.074 INFO:teuthology.orchestra.run.smithi050.stderr: 93.1% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-26T21:52:37.084 INFO:teuthology.orchestra.run.smithi046.stderr: 93.2% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-26T21:52:37.086 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-04-26T21:52:37.086 INFO:teuthology.orchestra.run.smithi046.stderr:real 0m1.184s 2024-04-26T21:52:37.087 INFO:teuthology.orchestra.run.smithi046.stderr:user 0m4.563s 2024-04-26T21:52:37.087 INFO:teuthology.orchestra.run.smithi046.stderr:sys 0m0.177s 2024-04-26T21:52:37.179 INFO:teuthology.orchestra.run.smithi050.stderr: 93.0% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-26T21:52:37.268 INFO:teuthology.orchestra.run.smithi142.stderr: 92.9% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-26T21:52:37.271 INFO:teuthology.orchestra.run.smithi142.stderr: 2024-04-26T21:52:37.271 INFO:teuthology.orchestra.run.smithi142.stderr:real 0m1.366s 2024-04-26T21:52:37.271 INFO:teuthology.orchestra.run.smithi142.stderr:user 0m4.384s 2024-04-26T21:52:37.271 INFO:teuthology.orchestra.run.smithi142.stderr:sys 0m0.176s 2024-04-26T21:52:37.301 INFO:teuthology.orchestra.run.smithi050.stderr: 92.8% -- replaced with /var/log/ceph/36c7013a-0413-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-26T21:52:37.303 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-04-26T21:52:37.304 INFO:teuthology.orchestra.run.smithi050.stderr:real 0m1.398s 2024-04-26T21:52:37.304 INFO:teuthology.orchestra.run.smithi050.stderr:user 0m5.042s 2024-04-26T21:52:37.304 INFO:teuthology.orchestra.run.smithi050.stderr:sys 0m0.186s 2024-04-26T21:52:37.304 INFO:tasks.cephadm:Archiving logs... 2024-04-26T21:52:37.304 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675215/remote/smithi046/log 2024-04-26T21:52:37.305 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T21:52:37.948 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675215/remote/smithi050/log 2024-04-26T21:52:37.949 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T21:52:38.673 DEBUG:teuthology.misc:Transferring archived files from smithi142:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675215/remote/smithi142/log 2024-04-26T21:52:38.674 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T21:52:39.336 INFO:tasks.cephadm:Removing cluster... 2024-04-26T21:52:39.337 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 36c7013a-0413-11ef-bc93-c7b262605968 --force 2024-04-26T21:52:39.542 INFO:teuthology.orchestra.run.smithi046.stdout:Deleting cluster with fsid: 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:52:39.808 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 36c7013a-0413-11ef-bc93-c7b262605968 --force 2024-04-26T21:52:40.008 INFO:teuthology.orchestra.run.smithi050.stdout:Deleting cluster with fsid: 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:52:40.282 DEBUG:teuthology.orchestra.run.smithi142:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 36c7013a-0413-11ef-bc93-c7b262605968 --force 2024-04-26T21:52:40.471 INFO:teuthology.orchestra.run.smithi142.stdout:Deleting cluster with fsid: 36c7013a-0413-11ef-bc93-c7b262605968 2024-04-26T21:52:40.967 INFO:tasks.cephadm:Removing cephadm ... 2024-04-26T21:52:40.967 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T21:52:40.986 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T21:52:41.005 DEBUG:teuthology.orchestra.run.smithi142:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T21:52:41.020 INFO:tasks.cephadm:Teardown complete 2024-04-26T21:52:41.020 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-26T21:52:41.035 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-26T21:52:41.035 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T21:52:41.038 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T21:52:41.047 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T21:52:41.142 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-26T21:52:41.151 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-26T21:52:41.151 DEBUG:teuthology.orchestra.run.smithi046:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T21:52:41.153 DEBUG:teuthology.orchestra.run.smithi050:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T21:52:41.156 DEBUG:teuthology.orchestra.run.smithi142:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T21:52:41.168 INFO:teuthology.orchestra.run.smithi046.stderr:bash: line 1: ntpq: command not found 2024-04-26T21:52:41.170 INFO:teuthology.orchestra.run.smithi142.stderr:bash: line 1: ntpq: command not found 2024-04-26T21:52:41.170 INFO:teuthology.orchestra.run.smithi050.stderr:bash: line 1: ntpq: command not found 2024-04-26T21:52:41.173 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T21:52:41.173 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2024-04-26T21:52:41.173 INFO:teuthology.orchestra.run.smithi046.stdout:^* hv01.front.sepia.ceph.com 3 6 377 58 +1385us[+1316us] +/- 53ms 2024-04-26T21:52:41.173 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 56 -478us[ -478us] +/- 58ms 2024-04-26T21:52:41.173 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 55 -1537us[-1537us] +/- 73ms 2024-04-26T21:52:41.173 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:52:41.175 INFO:teuthology.orchestra.run.smithi142.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T21:52:41.175 INFO:teuthology.orchestra.run.smithi142.stdout:=============================================================================== 2024-04-26T21:52:41.175 INFO:teuthology.orchestra.run.smithi142.stdout:^* hv01.front.sepia.ceph.com 3 6 377 56 +1370us[+1338us] +/- 53ms 2024-04-26T21:52:41.175 INFO:teuthology.orchestra.run.smithi142.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 56 -491us[ -523us] +/- 58ms 2024-04-26T21:52:41.175 INFO:teuthology.orchestra.run.smithi142.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 57 -1549us[-1582us] +/- 73ms 2024-04-26T21:52:41.175 INFO:teuthology.orchestra.run.smithi142.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:52:41.176 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T21:52:41.177 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2024-04-26T21:52:41.177 INFO:teuthology.orchestra.run.smithi050.stdout:^* hv01.front.sepia.ceph.com 3 6 377 58 +1286us[+1290us] +/- 53ms 2024-04-26T21:52:41.177 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 54 -563us[ -563us] +/- 58ms 2024-04-26T21:52:41.177 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 -1631us[-1631us] +/- 73ms 2024-04-26T21:52:41.177 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:52:41.177 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-26T21:52:41.187 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-26T21:52:41.187 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-26T21:52:41.228 DEBUG:teuthology.orchestra.run.smithi046:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T21:52:41.308 DEBUG:teuthology.orchestra.run.smithi050:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T21:52:41.378 DEBUG:teuthology.orchestra.run.smithi142:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T21:52:41.442 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T21:52:41.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T21:52:41.472 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T21:52:41.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T21:52:41.502 DEBUG:teuthology.orchestra.run.smithi142:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T21:52:41.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T21:52:41.529 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-26T21:52:41.539 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-26T21:52:41.549 INFO:teuthology.task.internal:Duration was 2107.134325 seconds 2024-04-26T21:52:41.549 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-26T21:52:41.558 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-26T21:52:41.559 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T21:52:41.561 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T21:52:41.564 DEBUG:teuthology.orchestra.run.smithi142:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T21:52:41.600 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T21:52:41.602 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T21:52:41.606 INFO:teuthology.orchestra.run.smithi142.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T21:52:42.075 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-26T21:52:42.075 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi046.front.sepia.ceph.com 2024-04-26T21:52:42.076 DEBUG:teuthology.orchestra.run.smithi046:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T21:52:42.098 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi050.front.sepia.ceph.com 2024-04-26T21:52:42.098 DEBUG:teuthology.orchestra.run.smithi050:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T21:52:42.127 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi142.front.sepia.ceph.com 2024-04-26T21:52:42.127 DEBUG:teuthology.orchestra.run.smithi142:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T21:52:42.152 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-26T21:52:42.152 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T21:52:42.155 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T21:52:42.169 DEBUG:teuthology.orchestra.run.smithi142:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T21:52:42.216 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-26T21:52:42.217 DEBUG:teuthology.orchestra.run.smithi046:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T21:52:42.223 DEBUG:teuthology.orchestra.run.smithi050:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T21:52:42.239 DEBUG:teuthology.orchestra.run.smithi142:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T21:52:42.589 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-26T21:52:42.600 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-26T21:52:42.600 DEBUG:teuthology.orchestra.run.smithi046:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T21:52:42.624 DEBUG:teuthology.orchestra.run.smithi050:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T21:52:42.650 DEBUG:teuthology.orchestra.run.smithi142:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T21:52:42.674 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-26T21:52:42.685 DEBUG:teuthology.orchestra.run.smithi046:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T21:52:42.688 DEBUG:teuthology.orchestra.run.smithi050:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T21:52:42.692 DEBUG:teuthology.orchestra.run.smithi142:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T21:52:42.708 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = core 2024-04-26T21:52:42.713 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = core 2024-04-26T21:52:42.737 INFO:teuthology.orchestra.run.smithi142.stdout:kernel.core_pattern = core 2024-04-26T21:52:42.769 DEBUG:teuthology.orchestra.run.smithi046:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T21:52:42.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T21:52:42.783 DEBUG:teuthology.orchestra.run.smithi050:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T21:52:42.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T21:52:42.797 DEBUG:teuthology.orchestra.run.smithi142:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T21:52:42.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T21:52:42.823 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-26T21:52:42.833 INFO:teuthology.task.internal:Transferring archived files... 2024-04-26T21:52:42.834 DEBUG:teuthology.misc:Transferring archived files from smithi046:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675215/remote/smithi046 2024-04-26T21:52:42.834 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T21:52:42.874 DEBUG:teuthology.misc:Transferring archived files from smithi050:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675215/remote/smithi050 2024-04-26T21:52:42.875 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T21:52:42.910 DEBUG:teuthology.misc:Transferring archived files from smithi142:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675215/remote/smithi142 2024-04-26T21:52:42.914 DEBUG:teuthology.orchestra.run.smithi142:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T21:52:42.948 INFO:teuthology.task.internal:Removing archive directory... 2024-04-26T21:52:42.949 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T21:52:42.952 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T21:52:42.954 DEBUG:teuthology.orchestra.run.smithi142:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T21:52:42.997 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-26T21:52:43.010 INFO:teuthology.task.internal:Not uploading archives. 2024-04-26T21:52:43.010 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-26T21:52:43.021 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-26T21:52:43.021 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T21:52:43.024 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T21:52:43.026 DEBUG:teuthology.orchestra.run.smithi142:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T21:52:43.036 INFO:teuthology.orchestra.run.smithi046.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 21:52 /home/ubuntu/cephtest 2024-04-26T21:52:43.039 INFO:teuthology.orchestra.run.smithi050.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 21:52 /home/ubuntu/cephtest 2024-04-26T21:52:43.052 INFO:teuthology.orchestra.run.smithi142.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 21:52 /home/ubuntu/cephtest 2024-04-26T21:52:43.053 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-26T21:52:43.064 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-26T21:52:43.086 INFO:teuthology.nuke:Checking targets against current locks 2024-04-26T21:52:43.111 DEBUG:teuthology.nuke:shortname: smithi046 2024-04-26T21:52:43.112 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T21:52:43.133 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675215', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 21:11:56.421096', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCPitY8pbKyqnD3letyL81Rd7976Rf9F5cF0n8DCQvpjF5yIjE8lfMg/4hlR0BEwRKWZhD8Q3ZPYahzPkbuVOT8='} 2024-04-26T21:52:43.137 DEBUG:teuthology.nuke:shortname: smithi050 2024-04-26T21:52:43.138 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T21:52:43.159 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675215', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 21:11:56.418044', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDKRkScLIIgerq406FE4WuJpgbkzbP5Zy3nIcvvak+ZGp4USLJ0Z62WLUR0fFPmieX3Tr0lxlVADn+tqGdyh8FY='} 2024-04-26T21:52:43.166 DEBUG:teuthology.nuke:shortname: smithi142 2024-04-26T21:52:43.166 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T21:52:43.191 INFO:teuthology.orchestra.console.smithi046:Power off 2024-04-26T21:52:43.191 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T21:52:43.201 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi142.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675215', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 21:11:56.419723', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDCpk9CFgwBtbGIYebYdy+7xw10lkXXTiTBbhGZb8esOlDP4FDWEYDXifKfcaGi7jsJKmgJOrGfCuaF75MGtUvQ='} 2024-04-26T21:52:43.216 DEBUG:teuthology.orchestra.console.smithi046:power off output: Chassis Power Control: Down/Off 2024-04-26T21:52:43.216 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T21:52:43.232 INFO:teuthology.orchestra.console.smithi050:Power off 2024-04-26T21:52:43.232 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T21:52:43.244 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2024-04-26T21:52:43.254 DEBUG:teuthology.orchestra.console.smithi050:power off output: Chassis Power Control: Down/Off 2024-04-26T21:52:43.254 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T21:52:43.272 INFO:teuthology.orchestra.console.smithi142:Power off 2024-04-26T21:52:43.272 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T21:52:43.282 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is on 2024-04-26T21:52:43.296 DEBUG:teuthology.orchestra.console.smithi142:power off output: Chassis Power Control: Down/Off 2024-04-26T21:52:43.297 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T21:52:43.321 DEBUG:teuthology.orchestra.console.smithi142:check power output: Chassis Power is on 2024-04-26T21:52:47.245 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T21:52:47.282 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T21:52:47.323 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T21:52:47.357 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2024-04-26T21:52:47.392 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is on 2024-04-26T21:52:47.431 DEBUG:teuthology.orchestra.console.smithi142:check power output: Chassis Power is on 2024-04-26T21:52:51.361 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T21:52:51.392 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T21:52:51.432 DEBUG:teuthology.orchestra.console.smithi142:pexpect command: ipmitool -H smithi142.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T21:52:51.472 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is off 2024-04-26T21:52:51.503 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is off 2024-04-26T21:52:51.542 DEBUG:teuthology.orchestra.console.smithi142:check power output: Chassis Power is off 2024-04-26T21:52:51.573 INFO:teuthology.orchestra.console.smithi046:Power off completed 2024-04-26T21:52:51.604 INFO:teuthology.orchestra.console.smithi050:Power off completed 2024-04-26T21:52:51.642 INFO:teuthology.orchestra.console.smithi142:Power off completed 2024-04-26T21:52:51.749 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} duration: 2107.1343245506287 failure_reason: 'Command failed on smithi046 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c ''ceph fs subvolumegroup pin cephfs qa random 0.10''"' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=a76c6789f7584bd494aa1f704fd701ab status: fail success: false 2024-04-26T21:52:51.749 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T21:52:51.871 INFO:teuthology.run:FAIL